AI
Why English-Only Testing Misses Arabic AI Failures

English-only evaluations can leave Arabic deployment risks untested. Eng. Mahmoud J. M. Alharazin examines dialects, Arabizi and cross-language retrieval. He sets out a practical approach to evidence, review and release decisions.
[For more news, click here]
By Eng. Mahmoud J. M. Alharazin, Managing Director, MAGNET INNOVATE LTD
An AI system can pass an English evaluation and still fail when it reaches Arabic-speaking users.
This is not simply a translation problem. Arabic deployment introduces a different assurance surface: Modern Standard Arabic, national and local dialects, Arabizi, code-switching, mixed-direction documents, translated policy wording, and records that move between Arabic and English. If the test plan treats Arabic as one translated copy of the English test set, important failures can remain invisible until production.
For GCC organisations deploying AI in customer service, internal search, compliance, HR, healthcare, finance or public services, language coverage should therefore be treated as a control requirement—not a final localisation check.
The First Mistake: Treating Arabic As One Test Case
Arabic is not one operational language condition. A user may write formal Arabic, a Gulf dialect, English mixed with Arabic, or Arabizi using Latin characters and numbers. A retrieved document may contain an English product name inside Arabic text, while the governing policy exists only in formal Arabic.
Research on code-switched Arabic describes a complex environment involving Modern Standard Arabic, dialects and several European languages. Research on Arabizi also shows that mixed Arabizi, English, French and Arabic-script text requires word-level language identification rather than simple document-level classification.
The assurance consequence is straightforward: a model that performs well on formal Arabic may not behave consistently on dialect, Arabizi or mixed-language input. “Arabic tested” is therefore too vague to be meaningful.
A better release statement is specific: which dialects, scripts, task types and risk conditions were tested, and what evidence supports the decision?
Five Failure Modes That English Testing Can Hide
1. Instruction drift
A translated instruction can preserve the general topic while changing a condition, exception or prohibition. In low-risk content this may be inconvenient. In finance, security, eligibility or compliance workflows, one changed qualifier can alter the decision.
Teams should test the original Arabic instruction and the translated version separately. Translation should not silently become the authoritative record.
2. Dialect mismatch
Users rarely speak to systems in policy language. A Saudi, Emirati, Egyptian, Levantine or Moroccan phrasing may express the same intent differently. Recent Arabic benchmark research continues to report variation across dialects, while dialectal reading-comprehension work highlights gaps between semantic answer quality and dialect fidelity.
Passing Modern Standard Arabic tests does not prove that the system understands the language actually used in a service channel.
3. Arabizi and code-switching
A support message may include Arabic meaning written in Latin characters, English technical terms and numbers used as Arabic letters. A security ticket might move between Arabic, Arabizi and English in a single paragraph.
The system can misclassify intent, miss a named entity, retrieve the wrong record or apply the wrong safety rule. These are not edge cases if the target users communicate this way.
4. Mixed-language retrieval failure
Retrieval-augmented systems can answer fluently while retrieving evidence in the wrong language, from the wrong version or from a translated summary that omits an exception.
For example, an Arabic question may retrieve an English FAQ instead of the controlling Arabic policy. The generated answer may sound correct even when the evidence hierarchy is wrong.
Retrieval testing should therefore measure more than answer quality. It should record which source was retrieved, its language, version, authority and whether the answer preserved the source meaning.
5. Hallucination that survives translation
Arabic hallucination research shows why fluent output is not enough. A model can generate a polished Arabic answer containing factual errors or unsupported additions. Translating that answer back into English may make review easier, but it does not prove that the original Arabic output was faithful.
The reviewer must inspect the original-language output against the original evidence, especially for high-impact decisions.
A Practical Arabic AI Assurance Method
Organisations do not need an enormous laboratory to improve this. They need a traceable test structure.
Step 1: Define the language coverage matrix
List the actual conditions the system will face:
- Modern Standard Arabic
- target GCC dialects
- Arabizi
- Arabic-English code-switching
- mixed-direction documents
- Arabic questions against English sources
- English questions against Arabic sources
Map each condition to real tasks such as classification, extraction, retrieval, summarisation, recommendation or decision support.
Step 2: Preserve original-language evidence
Keep the original prompt, response and retrieved source. A translated copy can support review, but it should not replace the original evidence.
The test record should show the language, dialect or script condition, model and system version, retrieval results, reviewer decision and any accepted limitation.
Step 3: Separate the components
Do not evaluate the whole system as one black box. Test:
- intent recognition
- translation
- retrieval
- generation
- policy application
- refusal and escalation behaviour
This makes it possible to locate the failure. Otherwise, a wrong answer may be blamed on the model when the actual cause was a translation layer or an outdated document index.
Step 4: Use risk-based acceptance criteria
A chatbot greeting and a financial eligibility decision should not share the same threshold.
For higher-impact uses, require stronger evidence, bilingual human review, explicit escalation rules and a reliable way to reproduce the result. NIST's Generative AI Profile places testing, evaluation, verification and validation within an ongoing risk-management process rather than a one-time launch exercise.
Step 5: Review failures by language condition
A single average score can hide a weak dialect or task. Report results separately by language, dialect, script and risk category.
If formal Arabic passes but Arabizi intent classification fails, the release decision should state that limitation clearly. The product may need a restricted scope, fallback workflow or additional human review.
The Governance Question
The most important question is not: “Does the model support Arabic?”
It is: “What Arabic conditions can this system handle reliably, for which tasks, at what level of risk, and what happens when it is uncertain?”
That question changes procurement, quality assurance and governance. Vendors should be asked for evidence by language condition, not a checkmark beside “Arabic.” Internal teams should keep original-language test records. Release owners should know which weaknesses are accepted and which trigger human escalation.
English-only evaluation is attractive because it is easier to standardise. But for organisations serving Arabic-speaking users, it creates a false sense of assurance. The system may pass the test environment while failing the language environment in which the real decision is made.
Arabic AI assurance is not an optional localisation step. It is part of system quality, security and accountability.
Sources
NIST, Code-switching survey, Arabizi study, AraDiCE, AraHalluEval, EDRAC
About the Author:
Eng. Mahmoud J. M. Alharazin is Managing Director of MAGNET INNOVATE LTD, a UK-registered technology and business consultancy. His work spans cybersecurity, quality assurance, AI governance, information systems and technical project management.





