Your Multilingual AI Passed. Which Languages Failed?

A multilingual assistant can pass its release test while failing the people who use it least often. If most evaluation requests are in English, good English answers dominate the total. A smaller language can lose crucial qualifications, retrieve the wrong evidence or receive unnecessary refusals without moving the headline score very much.

The decision is not whether a model “supports” a language. It is which tasks the complete service may perform in that language, with which evidence and fallback. Approving a multilingual model globally is a much broader claim than approving a particular customer journey.

This guide develops a release method for text-based assistants and retrieval-augmented generation. The cited research establishes benchmark designs and their limits; the test matrix, arithmetic example and operating rules are AIEngine's engineering analysis, not results from an AIEngine deployment. Sources were checked on 14 September 2026.

Replace the language list with a service claim

Start with an observable promise: a person asking about a documented procedure can obtain the correct answer, its relevant qualification and a usable source reference in their chosen language. That promise includes retrieval, generation, uncertainty handling and access to help. Fluent output alone does not establish it.

Google's 2023 XTREME-UP research evaluates 88 under-represented languages across nine user-centred technologies, including recognition, translation and information access. Its importance here is methodological: capabilities are tied to tasks, not reduced to one language-count claim. Success on one benchmark task cannot establish performance on a different business workflow.

Define the unit of approval as a language-and-task combination, then add distinctions that materially affect the journey. Input language, evidence language and required response language may differ. A Polish question answered from English policy documents is not the same retrieval problem as a Polish question answered from Polish documents.

Record script, locale, channel and meaningful language mixing where relevant. Do not infer nationality or legal jurisdiction from the language a person uses. Nor should language detection silently override an explicit preference: a short message containing a product name may provide little reliable evidence about the user's intended language.

Our localisation guide covers preparing and reviewing translated product content. The narrower concern here is whether a running AI service has enough evidence to perform an approved task for each language population.

Keep paired tests and native-authored tests separate

Two evaluation sets answer different questions. Matched versions of the same case help isolate changes associated with language. Independently authored cases help establish whether the evaluation resembles what people actually ask.

Meta's Belebele benchmark, published at ACL in 2024, provides parallel multiple-choice reading-comprehension data across 122 language variants. Parallelism supports direct comparison on aligned material. But choosing among four answers to a supplied passage is not an end-to-end test of open-ended assistance, retrieval or tool use.

By contrast, Google's 2020 TyDi QA study collected questions in 11 languages without translating them, and question writers did not see the answers. That design targets a different distribution of information-seeking behaviour. It does not make every native-authored dataset representative, but it explains why an English test set translated repeatedly is an incomplete launch test.

Use paired cases for controlled comparisons. Have competent reviewers confirm that each version preserves the relevant facts, ambiguity and difficulty; an easier translation invalidates the comparison. Group all language versions of one case together when separating development and held-out evaluation, so the same underlying scenario does not cross the boundary unnoticed.

Use a second, separately reported set written directly by target-language users or qualified contributors working from realistic needs. Preserve ordinary phrasing and locally meaningful references rather than polishing everything into formal prose. Record how contributors were recruited and which users remain absent. Combining both sets into one score would erase the reason for collecting them separately.

Make the language matrix small enough to inspect

An exhaustive combination of every language, intent, script, length and channel is rarely feasible. Start from the actual service promise and identify intersections where failure would change the outcome. Allocate evidence deliberately, including low-volume but consequential journeys.

For each approved combination, include:

• An ordinary answerable request using the phrasing users actually employ.
• A request whose answer depends on a qualification or exception.
• A request for which the permitted evidence contains no answer.
• A near-match document that discusses the subject but does not support the answer.
• A language switch between turns, including an explicit response-language preference.
• Realistic mixed-language text, transliteration or regional variants where users need them.
• A long input that exercises the deployed truncation and response limits.
• A request that must reach a person, with the handoff tested in that language.

These are coverage obligations, not a suggestion that eight examples prove readiness. Attach expected behaviour and severity to each case. Protect confidential material through permitted fixtures and minimisation; the need for representative language does not justify copying unrestricted customer transcripts into a benchmark.

Keep a separate count for cases the team cannot judge. An unreviewed language is an evidence gap, not an implied pass because another language is related to it. If qualified reviewers are unavailable, narrow the automated claim and preserve a usable human route.

Locate the failure before changing the model

A multilingual answer can fail because the correct evidence never reached the generator. Changing the answer model may therefore leave the underlying problem untouched.

The 2023 MIRACL dataset evaluates retrieval across 18 languages with native-speaker relevance judgements. Importantly, its specified task is monolingual retrieval: query and corpus share a language. A MIRACL result should not be presented as proof that a production service can retrieve English evidence from a Welsh question.

Measure retrieval against judged relevant passages for each intended input/evidence language pairing. Inspect whether preprocessing, query translation, the embedding model or ranking changed which passages appeared. Record evidence identifiers and versions, not only the generated answer.

Then supply the generator with a fixed, verified evidence bundle. If this controlled test passes while the full pipeline fails, retrieval or its surrounding transformations deserve investigation. If both fail, inspect interpretation, instruction following and response construction. This is diagnostic separation, not proof of a single root cause.

Also inspect language-specific operational conditions. Measure actual token counts and truncation events rather than assuming equal character counts create equivalent workloads. Record complete-answer latency and timeouts for the deployed configuration. A language that regularly exceeds a limit may receive a degraded service even when its completed answers score well.

The retrieval-quality audit supplies the broader evidence-tracing method. Extend its case labels with the language pairing instead of inventing a single multilingual relevance score.

Test answerability, not just correct answers

An assistant must recognise both when evidence supports an answer and when it does not. Testing only answerable questions rewards a service that always produces something plausible. Testing only absent answers can reward a service that refuses useful work.

The November 2024 NoMIRACL paper separates relevant and non-relevant passage sets across 18 languages and evaluates both mistaken assertions of relevance and failures to recognise relevant evidence. This is a useful evaluation structure; its historical model results are not a ranking of today's systems.

In an application test, report unsupported answers on unanswerable cases separately from unnecessary deferrals on answerable cases. Define what a correct deferral must contain: a truthful limitation, an appropriate next step and the requested language where feasible. A refusal in an unintelligible language is not a successful customer handoff.

Keep safety boundaries invariant across languages. Equivalent requests must not gain access to prohibited tools or private records after translation or language switching. Where authority is determined by account and task policy, enforce that policy outside the model. A language model's successful refusal test does not replace access control.

Review the evaluator in the language being judged

A translated answer can sound fluent while changing the action a reader should take. Separate task correctness, evidence support, critical omissions, language adherence and usability in the scoring rubric. A cosmetic error and a reversed exception should not receive interchangeable deductions.

Have reviewers work from the original request and evidence, with the competence needed for both language and domain. Blind the candidate identity where practical. Double-review a declared subset and every disputed critical error; preserve initial judgements and the adjudicated reason. Disagreement may reveal an ambiguous source policy rather than a model defect.

If an automated judge is used, calibrate it separately for each claimed language and task against human-reviewed cases. Measure false passes on serious errors and false failures on acceptable variants. Back-translating every answer into English before judging inserts another model and can conceal which component changed the meaning. It can assist inspection, but should not become the only evidence.

Do not assume a judge's overall agreement transfers to an under-sampled language. The LLM-as-a-judge calibration guide explains how to establish the evaluator's own limits before relying on its scores.

See what an average can hide

Consider an explicitly synthetic evaluation, not a claim about any model or language. Suppose a service reports these results under the same binary task-pass rubric:

A hypothetical aggregate threshold of 90% would pass this service. Yet the combined result is dominated by English. Giving the three language rates equal weight produces approximately 71.3%, which tells a different story but still does not decide whether any individual journey is acceptable.

Neither weighting is inherently fraudulent. A traffic-weighted result can describe expected overall performance under a specified traffic mix. An equal-language average can make smaller populations more visible. Both must sit beside per-language counts, rates, severity and uncertainty; neither should overrule a critical failure in an approved journey.

The 20 Welsh cases also support much less precise estimation than the 900 English cases. More repeated generations of the same 20 cases would measure run variation, not create new user situations. Use the evaluation sample-size guide to predeclare independent units, uncertainty intervals and comparison rules.

NIST's AI RMF Playbook, MEASURE 2.11 recommends context-specific analysis across and within groups, attention to small groups, and defined actions when disparities exceed tolerances. This is risk-management guidance, not a universal numerical language-parity threshold or a legal compliance certificate.

Make a release decision for each supported journey

Set criteria before examining candidate results. Require an absolute quality floor appropriate to the task, an allowed regression margin against the approved baseline, a separate critical-error rule and sufficient evidence for the language population. Equal performance is not enough if every language performs poorly.

Use three explicit outcomes: approved for the declared task; constrained to a narrower, tested route; or insufficient evidence for automated handling. A constraint must be operationally real. “Human review required” is not a control if no reviewer can read the language or no service capacity exists.

Keep the release record compact: the language and task boundary, evaluation-set provenance, case counts, critical findings, uncertainty, retrieval configuration, model and judge revisions, reviewer competence, fallback test and approving owner. Include unresolved disagreements and excluded populations.

Approval must not turn into service denial. Where an automated route is withheld, provide an accessible alternative capable of completing the task. Monitor whether that alternative creates disproportionate delay or abandonment. An English-only error message is not a meaningful fallback for everyone.

Watch the claim after launch

Track task success, unsupported answers, unnecessary deferrals, language-preference violations, failed handoffs, complete-request latency and complaints by approved language-and-task combination. Keep denominators visible. A low complaint count can reflect limited access or inability to report a problem rather than a satisfactory service.

Re-test when the model, retriever, query-translation step, tokenizer, prompt, source corpus or judge changes. Revisit the coverage matrix when users introduce new scripts, mixed-language patterns or tasks. Preserve the old result as evidence for its original boundary.

The useful release statement is deliberately specific: this service can perform these tasks for these language populations under these tested conditions. A high multilingual average may accompany that statement. It cannot stand in for it.