mlsanityData to 14 Oct 2026

ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer

arXiv 2502.21228 ↗First posted 28 Feb 2025cs.CL cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

To achieve equitable performance across languages, large language models (LLMs) must be able to abstract knowledge beyond the language in which it was learnt. However, the current literature lacks reliable ways to measure LLMs' capability of such cross-lingual knowledge transfer. To that end, we present ECLeKTic, a multilingual closed-book QA dataset that Evaluates Cross-Lingual Knowledge Transfer in a simple, black-box manner. Concretely, we used the presence and absence of Wikipedia articles in 12 languages to detect pieces of information that were likely available during pre-training in one of the languages but not in the others. We curate ECLeKTic as a set of fact-seeking questions over this kind of information, in all the different languages. Therefore, in order to solve ECLeKTic the model is required to transfer knowledge between languages. We evaluated 8 LLMs and showed that current SOTA models struggle to effectively share knowledge across languages, even if they can predict the answer for questions in the language in which the knowledge was acquired.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

33 of 43 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  5. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  6. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  7. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  8. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  9. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  10. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  11. 2022A Review on Language Models as Knowledge Basescited by 157
  12. 2024How do Large Language Models Handle Multilingualism?cited by 150
21 more
  1. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  2. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  3. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  4. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  5. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 96
  6. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 73
  7. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  8. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  9. 2024Improving the Robustness of Large Language Models via Consistency Alignmentcited by 36
  10. 2022A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bankcited by 29
  11. 2025Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMscited by 22
  12. 2024Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMscited by 21
  13. 2023Separating form and meaning: Using self-consistency to quantify task understanding across multiple sensescited by 15
  14. 2023Exploring Format Consistency for Instruction Tuningcited by 14
  15. 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
  16. 2024MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Modelscited by 11
  17. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  18. 2024Cross-Dialect Information Retrieval: Information Access in Low-Resource and High-Variance Languagescited by 8
  19. 2022mOKB6: A Multilingual Open Knowledge Base Completion Benchmarkcited by 7
  20. 2025Graders should cheat: privileged information enables expert-level automated evaluationscited by 5
  21. 2024How Many Images Does It Take? Estimating Imitation Thresholds in Text-to-Image Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 13.

29 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,824
  2. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 21
  3. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 20
  4. 2025Steering Language Models with Weight Arithmeticcited by 20
  5. 2025Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiencescited by 15
  6. 2025Multilingual Performance Biases of Large Language Models in Educationcited by 9
  7. 2026Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistencycited by 6
  8. 2025OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literaturecited by 4
  9. 2025CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Datasetcited by 4
  10. 2025Rethinking Cross-lingual Gaps from a Statistical Viewpointcited by 4
  11. 2026Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMscited by 4
  12. 2025WikiGap: Promoting Epistemic Equity by Surfacing Knowledge Gaps Between English Wikipedia and other Language Editionscited by 3
17 more
  1. 2025PropMEND: Hypernetworks for Knowledge Propagation in LLMscited by 3
  2. 2025Beyond the Rosetta Stone: Unification Forces in Generalization Dynamicscited by 3
  3. 2026Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scriptscited by 3
  4. 2025Searching the Internet for Challenging Benchmarks at Scalecited by 2
  5. 2026Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Dropscited by 2
  6. 2025A Diagnostic Benchmark for Sweden-Related Factual Knowledgecited by 1
  7. 2026Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QAcited by 1
  8. 2025Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Transfer in Sense-Aware Taskscited by 0
  9. 2025Language over Content: Tracing Cultural Understanding in Multilingual Large Language Modelscited by 0
  10. 2026Grandes Modelos de Linguagem Multimodais (MLLMs): Da Teoria `a Pr'aticacited by 0
  11. 2026Language models struggle with compartmentalizationcited by 0
  12. 2026Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfercited by 0
  13. 2026Probing Factual Knowledge Transfer with Training Data Interventionscited by 0
  14. 2026Target-Language Generation in Multilingual Models: Activation Steering and Optimal Controlcited by 0
  15. 2026Why Pretraining Fails to Share Cross-Lingual Knowledgecited by 0
  16. 2026Large-scale factor analysis shows machine intelligence is only partially interpretablecited by 0
  17. 2026Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparitiescited by 0