mlsanityData to 7 Oct 2026

LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs

arXiv 2507.16809 ↗First posted 22 Jul 2025cs.CLcited by 5

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We propose LingBench++, a linguistically-informed benchmark and reasoning framework designed to evaluate large language models (LLMs) on complex linguistic tasks inspired by the International Linguistics Olympiad (IOL). Unlike prior benchmarks that focus solely on final answer accuracy, LingBench++ provides structured reasoning traces, stepwise evaluation protocols, and rich typological metadata across over 90 low-resource and cross-cultural languages. We further develop a multi-agent architecture integrating grammatical knowledge retrieval, tool-augmented reasoning, and deliberate hypothesis testing. Through systematic comparisons of baseline and our proposed agentic models, we demonstrate that models equipped with external knowledge sources and iterative reasoning outperform single-pass approaches in both accuracy and interpretability. LingBench++ offers a comprehensive foundation for advancing linguistically grounded, culturally informed, and cognitively plausible reasoning in LLMs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 33 of its 48 references that are in the map. First posted on arXiv 22 Jul 2025.

Builds on

33 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Reasoning Over the Glyphs: Evaluation of LLM's Decipherment of Rare Scripts29 Jan
  2. 2025IOLBENCH: Benchmarking LLMs on Linguistic Reasoning8 Jan
  3. 2024Linguini: A benchmark for language-agnostic linguistic reasoning18 Sep
  4. 2024modeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models24 Jun
  5. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages10 Jun

Other references · 28

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  10. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  11. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  12. 2019How multilingual is Multilingual BERT?cited by 642
16 more
  1. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  2. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  3. 2023Large Language Model Guided Tree-of-Thoughtcited by 218
  4. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  5. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  6. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  7. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
  8. 2024Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Taskscited by 30
  9. 2024Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?cited by 23
  10. 2020PuzzLing Machines: A Challenge on Learning From Small Datacited by 15
  11. 2024Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoningcited by 13
  12. 2023Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Casescited by 12
  13. 2025Chain-of-Tools: Utilizing Massive Unseen Tools in the CoT Reasoning of Frozen Language Modelscited by 9
  14. 2021Sample-efficient Linguistic Generalizations through Program Synthesis: Experiments with Phonology Problemscited by 4
  15. 2024Semisupervised Neural Proto-Language Reconstructioncited by 3
  16. 2024Can Large Language Models Code Like a Linguist?: A Case Study in Low Resource Sound Law Inductioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlangcited by 3
  2. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  3. 2026Kinship Data Benchmark for Multi-hop Reasoningcited by 1
  4. 2026Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languagescited by 0
  5. 2026Think Through a Bottleneck: Hourglass Reasoning for Rigorous Inductioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Cross-Lingual Consistency: A Novel Inference Framework for Advancing Reasoning in Large Language Models
  2. 2025LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models
  3. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  4. 2025IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluation
  5. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
  6. 2025LingGym: How Far Are LLMs from Thinking Like Field Linguists?
  7. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  8. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMs