mlsanityData to 14 Oct 2026

ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation

arXiv 2607.23058 ↗First posted 25 Jul 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Multilingual reasoning evaluation overwhelmingly relies on translating English benchmarks, a practice that introduces linguistic artifacts and fails to test culturally-grounded reasoning. We introduce ADAGE (Analogical Difficulty-by-design Assessment for Grounded Evaluation), a language-agnostic pipeline that combines native-speaker curation with LLM-assisted generation to construct challenging, translation-free benchmarks for abstract analogical reasoning. We validate ADAGE by constructing benchmarks for Arabic, Amharic, and Japanese. Evaluating 14 open-weight models, we find a consistent cultural reasoning gap: models that perform well on English proverb reasoning struggle substantially on all three native benchmarks, with accuracy dropping by 12--52 percentage points relative to English. We release the pipeline, all three benchmarks, and the full evaluation suite.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 28 of its 36 references that are in the map. First posted on arXiv 25 Jul 2026.

Builds on

28 of 36 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures28 Oct
  2. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs23 Jul
  3. 2025Qwen3 Technical Report14 May
  4. 2025A Survey of Multilingual Reasoning in Language Models13 Feb
  5. 2025IOLBENCH: Benchmarking LLMs on Linguistic Reasoning8 Jan
  6. 2024SynDARin: Synthesising Datasets for Automated Reasoning in Low-Resource Languages20 Jun

Other references · 22

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  8. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  9. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  10. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  11. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  12. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
10 more
  1. 2019Translationese in Machine Translation Evaluationcited by 38
  2. 2024CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeamingcited by 30
  3. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  4. 2025Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarkingcited by 18
  5. 2025MMATH: A Multilingual Benchmark for Mathematical Reasoningcited by 17
  6. 2024ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understandingcited by 15
  7. 2021ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understandingcited by 14
  8. 2024Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settingscited by 14
  9. 2025CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understandingcited by 14
  10. 2025ARB: A Comprehensive Arabic Multimodal Reasoning Benchmarkcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Inductive Linguistic Reasoning with Large Language Models
  2. 2023ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base
  3. 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
  4. 2024Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models
  5. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Models
  6. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs
  7. 2022E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning
  8. 2025IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation