mlsanityData to 7 Oct 2026

Linguini: A benchmark for language-agnostic linguistic reasoning

arXiv 2409.12126 ↗First posted 18 Sep 2024cs.CLcited by 17

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We propose a new benchmark to measure a language model's linguistic reasoning skills without relying on pre-existing language-specific knowledge. The test covers 894 questions grouped in 160 problems across 75 (mostly) extremely low-resource languages, extracted from the International Linguistic Olympiad corpus. To attain high accuracy on this benchmark, models don't need previous knowledge of the tested language, as all the information needed to solve the linguistic puzzle is presented in the context. We find that, while all analyzed models rank below 25% accuracy, there is a significant gap between open and closed models, with the best-performing proprietary model at 24.05% and the best-performing open model at 8.84%.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 35 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

20 of 35 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  2. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance26 May
  3. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-47 Apr
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity8 Feb

Other references · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2024Mixtral of Expertscited by 1,785
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  9. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  11. 2023Mathematical Capabilities of ChatGPTcited by 263
  12. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
4 more
  1. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  2. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  3. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  4. 2024Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptionscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 5.

17 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2025BIG-Bench Extra Hardcited by 123
  3. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 22
  4. 2025KoBALT: Korean Benchmark For Advanced Linguistic Taskscited by 12
  5. 2025ScholarEval: Research Idea Evaluation Grounded in Literaturecited by 10
  6. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMscited by 5
  7. 2025Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation Analysiscited by 4
  8. 2025LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscationcited by 3
  9. 2025UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?cited by 3
  10. 2026When Does Intrinsic Self-Correction Help? A Task-Sensitive Analysiscited by 3
  11. 2025Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzlescited by 2
  12. 2026Speaking of Language: Reflections on Metalanguage Research in NLPcited by 1
5 more
  1. 2026Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMscited by 1
  2. 2024MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Modelscited by 0
  3. 2025Testing the Limits of Machine Translation from One Bookcited by 0
  4. 2026The Linguistics Olympiads: Towards a New Corpus for Linguistics Research?cited by 0
  5. 2026Load-Bearing Context: The Question Damage Score for Evaluating Context Reliance in Linguistic Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
  2. 2024modeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models
  3. 2022Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
  4. 2025IOLBENCH: Benchmarking LLMs on Linguistic Reasoning
  5. 2025ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models
  6. 2026BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
  7. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
  8. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models