Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models

Overview: Active Cross-Lingual Failure Discovery

Cross-Lingual Pitfalls introduces search-based multilingual LLM evaluation. It actively discovers cross-lingual weaknesses: cases where a model answers a question correctly in English but incorrectly in a semantically checked target-language version.

Rather than only measuring average accuracy on a fixed multilingual test set, the method combines beam search and LLM-based simulation to find questions that expose an English–target-language performance gap. The released resource contains 6,713 bilingual pairs across 16 target languages, originating from MMLU, ARC, CommonsenseQA, TruthfulQA, and SciQ. This is a failure-focused collection, so its observed accuracy gaps describe the selected questions and experimental models—not average performance for all users or questions in those languages.

In the authors' words, the work makes three contributions:

1. "We present an efficient, precise methodology for identifying LLM cross-lingual weaknesses."
2. "we construct a novel, 16-language dataset with over 6,000 bilingual pairs to challenge cross-lingual capabilities."
3. "Extensive experiments ... quantitatively analyze the relationship between cross-lingual weaknesses and linguistic similarities and fine-tuning experiments demonstrate the potential for targeted cross-lingual improvement."

From Fixed Benchmarks to Active Failure Discovery

Fixed multilingual benchmarks estimate performance on their chosen questions; cross-lingual consistency tests whether a model handles corresponding questions similarly across languages. Cross-Lingual Pitfalls adds a complementary question: can we automatically find where that consistency breaks? Searching for English-correct, target-language-wrong pairs can reveal conditional failure regions that an aggregate score obscures.

The generation method can be run again to probe other model settings. The published 6,713-pair dataset is a fixed release, not a continuously changing benchmark. The work probes multilingual question answering and cross-lingual failure discovery; it is not a benchmark of cultural knowledge, code-switching, translation quality, safety, or agent tool use.

Abstract

Key Findings on the Searched Failure Cases

How We Search for Cross-Lingual Failures

Cross-Lingual Pitfalls uses beam-search perturbations and LLM-based simulation to find bilingual question pairs that expose English–target-language discrepancies. It is an active probing method that complements fixed multilingual benchmarks, rather than another estimate of average multilingual accuracy.

Starting from English questions drawn from MMLU, ARC, CommonsenseQA, TruthfulQA, and SciQ, the search proposes question variants, checks corresponding target-language versions, and scores candidates for a large English–target-language performance gap using LLM-based simulation. The paper evaluates the resulting pairs on ten models. A rerun can produce new probes, but the released 6,713 pairs do not change automatically.

Dataset: 6,713 Cross-Lingual Failure Probes

The fixed dataset contains 6,713 bilingual question pairs spanning 16 target languages, built from five English QA benchmarks (MMLU, ARC, CommonsenseQA, TruthfulQA, and SciQ). It is available on the Hugging Face Hub and in the GitHub repository. Its pairs were selected to expose failures, so comparisons on this resource should not be presented as representative language-wide accuracy.

6,713 bilingual English–target pairs · 16 languages · built from five English QA benchmarks · designed to expose cases where an LLM is correct in English but wrong in the target language · released on Hugging Face and GitHub · useful for evaluating cross-lingual consistency, multilingual robustness, and non-English failure modes.

Total across 16 languages: 6,713 bilingual (English ↔ target) pairs.

In the paper's evaluation on selected pairs, Amharic, Yoruba, Swahili, and Zulu show particularly large gaps. Counts above report dataset sizes only; per-language accuracy results and experimental details are in the paper.

From Failure Discovery to Multilingual Post-Training

Finding a failure can also identify a training opportunity. The paper tests targeted post-training with Chinese and French pairs, using supervised fine-tuning and simulated preference optimization on four models. It examines whether improvements carry over to other tested languages and finds transfer in some related or similar language settings. These experiments support a targeted route from diagnosis to improvement, without establishing a universal post-training recipe.

Frequently Asked Questions

When to Cite This Paper

Cross-Lingual Pitfalls is a useful reference when your work touches any of the following:

• search-based or active multilingual LLM evaluation;
• automatic discovery of cross-lingual weaknesses and consistency failures;
• English-correct, target-language-wrong cases that aggregate benchmark scores can obscure;
• failure-focused bilingual question sets across 16 target languages;
• targeted multilingual post-training informed by discovered failures.

A typical citation: Xu et al. (2025) introduce a search-based probe that automatically discovers English-correct, target-language-wrong questions, complementing fixed multilingual benchmarks with targeted cross-lingual failure discovery.

Resources

• arXiv:2505.18673 — abstract page (submitted 24 May 2025). HTML version.
• ACL Anthology — ACL 2025, Vienna, Austria, pages 8254–8284, DOI 10.18653/v1/2025.acl-long.404.
• Dataset on Hugging Face — the 16-language, 6,713-pair benchmark, loadable with the datasets library.
• Code & dataset on GitHub — MIT license; the 16-language dataset ships inside the repository.

BibTeX

@inproceedings{xu-etal-2025-cross,
    title = "Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models",
    author = "Xu, Zixiang  and Wang, Yanbo  and Huang, Yue  and Chen, Xiuying  and Zhao, Jieyu  and Jiang, Meng  and Zhang, Xiangliang",
    editor = "Che, Wanxiang  and Nabende, Joyce  and Shutova, Ekaterina  and Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul, year = "2025", address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.acl-long.404/",
    doi = "10.18653/v1/2025.acl-long.404",
    pages = "8254--8284", ISBN = "979-8-89176-251-0"
}

Prefer the preprint? An arXiv citation for 2505.18673 is also available from the arXiv page.