AI

Why English-Only Testing Misses Arabic AI Failures

Conceptual illustration of Arabic and Latin letterforms entering an AI verification prism with a gap in one data path.
AI-generated conceptual illustration. It does not depict a measured test or an actual system.

English-only evaluations can leave Arabic deployment risks untested. Eng. Mahmoud J. M. Alharazin examines dialects, Arabizi and cross-language retrieval. He sets out a practical approach to evidence, review and release decisions.

[For more news, click here]

By Eng. Mahmoud J. M. Alharazin, Managing Director, MAGNET INNOVATE LTD

An AI system can pass an English evaluation and still fail when it reaches Arabic-speaking users.

This is not simply a translation problem. Arabic deployment introduces a different assurance surface: Modern Standard Arabic, national and local dialects, Arabizi, code-switching, mixed-direction documents, translated policy wording, and records that move between Arabic and English. If the test plan treats Arabic as one translated copy of the English test set, important failures can remain invisible until production.

For GCC organisations deploying AI in customer service, internal search, compliance, HR, healthcare, finance or public services, language coverage should therefore be treated as a control requirement—not a final localisation check.

The First Mistake: Treating Arabic As One Test Case

Arabic is not one operational language condition. A user may write formal Arabic, a Gulf dialect, English mixed with Arabic, or Arabizi using Latin characters and numbers. A retrieved document may contain an English product name inside Arabic text, while the governing policy exists only in formal Arabic.

Research on code-switched Arabic describes a complex environment involving Modern Standard Arabic, dialects and several European languages. Research on Arabizi also shows that mixed Arabizi, English, French and Arabic-script text requires word-level language identification rather than simple document-level classification.

The assurance consequence is straightforward: a model that performs well on formal Arabic may not behave consistently on dialect, Arabizi or mixed-language input. “Arabic tested” is therefore too vague to be meaningful.

A better release statement is specific: which dialects, scripts, task types and risk conditions were tested, and what evidence supports the decision?

Five Failure Modes That English Testing Can Hide

1. Instruction drift

A translated instruction can preserve the general topic while changing a condition, exception or prohibition. In low-risk content this may be inconvenient. In finance, security, eligibility or compliance workflows, one changed qualifier can alter the decision.

Teams should test the original Arabic instruction and the translated version separately. Translation should not silently become the authoritative record.

2. Dialect mismatch

Users rarely speak to systems in policy language. A Saudi, Emirati, Egyptian, Levantine or Moroccan phrasing may express the same intent differently. Recent Arabic benchmark research continues to report variation across dialects, while dialectal reading-comprehension work highlights gaps between semantic answer quality and dialect fidelity.

Passing Modern Standard Arabic tests does not prove that the system understands the language actually used in a service channel.

3. Arabizi and code-switching

A support message may include Arabic meaning written in Latin characters, English technical terms and numbers used as Arabic letters. A security ticket might move between Arabic, Arabizi and English in a single paragraph.

The system can misclassify intent, miss a named entity, retrieve the wrong record or apply the wrong safety rule. These are not edge cases if the target users communicate this way.

4. Mixed-language retrieval failure

Retrieval-augmented systems can answer fluently while retrieving evidence in the wrong language, from the wrong version or from a translated summary that omits an exception.

For example, an Arabic question may retrieve an English FAQ instead of the controlling Arabic policy. The generated answer may sound correct even when the evidence hierarchy is wrong.

Retrieval testing should therefore measure more than answer quality. It should record which source was retrieved, its language, version, authority and whether the answer preserved the source meaning.

5. Hallucination that survives translation

Arabic hallucination research shows why fluent output is not enough. A model can generate a polished Arabic answer containing factual errors or unsupported additions. Translating that answer back into English may make review easier, but it does not prove that the original Arabic output was faithful.

The reviewer must inspect the original-language output against the original evidence, especially for high-impact decisions.

A Practical Arabic AI Assurance Method

Organisations do not need an enormous laboratory to improve this. They need a traceable test structure.

Step 1: Define the language coverage matrix

List the actual conditions the system will face:

  • Modern Standard Arabic
  • target GCC dialects
  • Arabizi
  • Arabic-English code-switching
  • mixed-direction documents
  • Arabic questions against English sources
  • English questions against Arabic sources

Map each condition to real tasks such as classification, extraction, retrieval, summarisation, recommendation or decision support.

Step 2: Preserve original-language evidence

Keep the original prompt, response and retrieved source. A translated copy can support review, but it should not replace the original evidence.

The test record should show the language, dialect or script condition, model and system version, retrieval results, reviewer decision and any accepted limitation.

Step 3: Separate the components

Do not evaluate the whole system as one black box. Test:

  • intent recognition
  • translation
  • retrieval
  • generation
  • policy application
  • refusal and escalation behaviour

This makes it possible to locate the failure. Otherwise, a wrong answer may be blamed on the model when the actual cause was a translation layer or an outdated document index.

Step 4: Use risk-based acceptance criteria

A chatbot greeting and a financial eligibility decision should not share the same threshold.

For higher-impact uses, require stronger evidence, bilingual human review, explicit escalation rules and a reliable way to reproduce the result. NIST's Generative AI Profile places testing, evaluation, verification and validation within an ongoing risk-management process rather than a one-time launch exercise.

Step 5: Review failures by language condition

A single average score can hide a weak dialect or task. Report results separately by language, dialect, script and risk category.

If formal Arabic passes but Arabizi intent classification fails, the release decision should state that limitation clearly. The product may need a restricted scope, fallback workflow or additional human review.

The Governance Question

The most important question is not: “Does the model support Arabic?”

It is: “What Arabic conditions can this system handle reliably, for which tasks, at what level of risk, and what happens when it is uncertain?”

That question changes procurement, quality assurance and governance. Vendors should be asked for evidence by language condition, not a checkmark beside “Arabic.” Internal teams should keep original-language test records. Release owners should know which weaknesses are accepted and which trigger human escalation.

English-only evaluation is attractive because it is easier to standardise. But for organisations serving Arabic-speaking users, it creates a false sense of assurance. The system may pass the test environment while failing the language environment in which the real decision is made.

Arabic AI assurance is not an optional localisation step. It is part of system quality, security and accountability.

Sources

NIST, Code-switching survey, Arabizi study, AraDiCE, AraHalluEval, EDRAC

About the Author:

Eng. Mahmoud J. M. Alharazin is Managing Director of MAGNET INNOVATE LTD, a UK-registered technology and business consultancy. His work spans cybersecurity, quality assurance, AI governance, information systems and technical project management.

Related Articles

Abu Dhabi Launches AI Tools to Rewrite Arabic Texts for Different Reading Levels
AI

Abu Dhabi Launches AI Tools to Rewrite Arabic Texts for Different Reading Levels

Raghav Sehgal, General Manager Middle East at Model ML
AI

Exclusive: Raghav Sehgal on the Shift Towards Agentic AI in Financial Services

us
AI

US offers $50 million to support Malaysia’s sovereign AI cloud initiative

AI generated image
AI

Jordan Is Building More Than an AI Campus. It Is Building the Stack Around It

Conceptual illustration of connected AI computing infrastructure in a financial district atrium.
AI

Anthropic IPO Puts AI's Business Model Under Market Scrutiny

Mitsubishi Heavy Industries and Sakura Internet sign MoU on sovereign AI infrastructure in Japan
AI

Mitsubishi Heavy Industries and Sakura Internet sign MoU on sovereign AI infrastructure in Japan