Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models

Overview (TL;DR)

Multilingual LLMs can answer a question correctly in English yet get the very same question wrong in another language. Cross-Lingual Pitfalls makes this measurable: across 16 languages, most models lose over 50% of their accuracy once a question leaves English—and the failures are systematic, not noise.

In practice this is a silent failure mode that standard, English-centric evaluation misses: a model can look near-perfect on English benchmarks yet collapse once the same question is posed in Hindi, Arabic, or Swahili—GPT-4o alone drops nearly 30% in Chinese. Cross-Lingual Pitfalls turns the search for these gaps into an automatic procedure. Instead of relying on a fixed, translated test set, it actively generates bilingual (English ↔ target-language) question pairs designed to maximize the accuracy gap, exposing weaknesses that static, English-centric evaluations tend to miss. The result is a reusable, 16-language dataset of 6,713 bilingual pairs (built from MMLU) and a cheap, scalable diagnostic that works even on state-of-the-art models such as GPT-4o and Claude-3.5-Sonnet.

In the authors' words, the work makes three contributions:

1. "We present an efficient, precise methodology for identifying LLM cross-lingual weaknesses."
2. "we construct a novel, 16-language dataset with over 6,000 bilingual pairs to challenge cross-lingual capabilities."
3. "Extensive experiments ... quantitatively analyze the relationship between cross-lingual weaknesses and linguistic similarities and fine-tuning experiments demonstrate the potential for targeted cross-lingual improvement."

Abstract

Key Findings & Why They Matter

How It Works (Method)

Cross-Lingual Pitfalls combines beam search with LLM-based simulation to generate adversarial bilingual question pairs that maximize the English-vs-target accuracy gap, going beyond static translated benchmarks such as MGSM or Global-MMLU.

Starting from MMLU questions, the method uses beam search as a perturbation and search strategy, while an LLM-based simulation stands in for the target model to estimate how a candidate question will be answered in English versus the target language. Each step keeps the variants that widen the English-vs-target accuracy gap, so the search converges on bilingual (English ↔ target-language) pairs where the model is right in English and wrong in the target language. Because the search is gap-maximizing and fully automatic, it produces a dynamic benchmark that adapts to each model instead of reusing one fixed, English-centric test set. There is no branded acronym; the work is referred to simply as Cross-Lingual Pitfalls.

Dataset: The Cross-Lingual Pitfalls Benchmark

The probing method yields a reusable resource: 6,713 bilingual question pairs spanning 16 languages, all built from MMLU and distributed inside the GitHub repository (there is no separate Hugging Face dataset). They cover a broad range of languages, with the widest cross-lingual gaps appearing in the lowest-resource languages.

6,713 bilingual English–target pairs · 16 languages · built from MMLU · designed to expose cases where an LLM is correct in English but wrong in the target language · released on GitHub · useful for evaluating cross-lingual consistency, multilingual robustness, and non-English failure modes.

Total across 16 languages: 6,713 bilingual (English ↔ target) pairs.

The low-resource African languages (Amharic, Yoruba, Swahili, Zulu) show the largest cross-lingual gaps. Counts are the number of bilingual (English ↔ target) pairs per language and sum to 6,713; these counts report dataset sizes only—per-language accuracy figures are in the paper.

Frequently Asked Questions

When to Cite This Paper

Cross-Lingual Pitfalls is a useful reference when your work touches any of the following:

• cross-lingual inconsistency in multilingual LLMs;
• automatic or adversarial probing of multilingual model failures;
• English-centric blind spots in LLM evaluation;
• low-resource language evaluation;
• dynamic benchmarks beyond static, translated test sets;
• language-family transfer in post-training or fine-tuning;
• cases where a model is correct in English but fails on a semantically equivalent non-English question.

A typical citation: Xu et al. (2025) show that multilingual LLMs can answer questions correctly in English while failing on semantically equivalent questions in target languages, exposing systematic cross-lingual weaknesses.

Resources

• arXiv:2505.18673 — abstract page (submitted 24 May 2025). HTML version.
• PDF — preprint full text.
• ACL Anthology — ACL 2025, Vienna, Austria, pages 8254–8284, DOI 10.18653/v1/2025.acl-long.404.
• Code & dataset on GitHub — MIT license; the 16-language dataset ships inside the repository.

BibTeX

@inproceedings{xu-etal-2025-cross,
    title = "Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models",
    author = "Xu, Zixiang  and Wang, Yanbo  and Huang, Yue  and Chen, Xiuying  and Zhao, Jieyu  and Jiang, Meng  and Zhang, Xiangliang",
    editor = "Che, Wanxiang  and Nabende, Joyce  and Shutova, Ekaterina  and Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul, year = "2025", address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.acl-long.404/",
    doi = "10.18653/v1/2025.acl-long.404",
    pages = "8254--8284", ISBN = "979-8-89176-251-0"
}

Prefer the preprint? An arXiv citation for 2505.18673 is also available from the arXiv page.