mlsanityData to 14 Oct 2026

LingGym: How Far Are LLMs from Thinking Like Field Linguists?

arXiv 2511.00343 ↗First posted 1 Nov 2025cs.CLcited by 6

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This paper introduces LingGym, a new benchmark that evaluates LLMs' capacity for meta-linguistic reasoning using Interlinear Glossed Text (IGT) and grammatical descriptions extracted from 18 typologically diverse reference grammars. Unlike previous work that focuses on specific downstream tasks, we assess whether LLMs can generalize linguistic inference across low-resource languages and structures not seen during training. We present a controlled evaluation task: Word-Gloss Inference, in which the model must infer a missing word and gloss from context using varying levels of linguistic information (e.g., glosses, grammatical explanations, translations). Our results show that incorporating structured linguistic cues leads to consistent improvements in reasoning performance across all models. This work highlights both the promise and current limitations of using LLMs for typologically informed linguistic analysis and low-resource language documentation.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 29 of its 54 references that are in the map. First posted on arXiv 1 Nov 2025.

Builds on

29 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages10 Jun

Other references · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2025Gemma 3 Technical Reportcited by 1,824
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  7. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  8. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  9. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  10. 2018Colorless green recurrent networks dream hierarchicallycited by 213
  11. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  12. 2019Assessing BERT's Syntactic Abilitiescited by 192
16 more
  1. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  2. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  3. 2020Universal Phone Recognition with a Multilingual Allophone Systemcited by 81
  4. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  5. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 75
  6. 2022ByT5 model for massively multilingual grapheme-to-phoneme conversioncited by 49
  7. 2017The Zero Resource Speech Challenge 2017cited by 45
  8. 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languagescited by 38
  9. 2024Teaching Large Language Models an Unseen Language on the Flycited by 30
  10. 2025ZIPA: A family of efficient models for multilingual phone recognitioncited by 19
  11. 2024Can we teach language models to gloss endangered languages?cited by 14
  12. 2024Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuningcited by 14
  13. 2024Can LLMs Help Create Grammar?: Automating Grammar Creation for Endangered Languages with In-Context Learningcited by 8
  14. 2024Multiple Sources are Better Than One: Incorporating External Knowledge in Low-Resource Glossingcited by 5
  15. 2024GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learningcited by 5
  16. 2025Developing multilingual speech synthesis system for Ojibwe, Mi'kmaq, and Maliseetcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Massively Multilingual Joint Segmentation and Glossingcited by 1
  2. 2026Speaking of Language: Reflections on Metalanguage Research in NLPcited by 1
  3. 2026Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languagescited by 0
  4. 2026Hybrid Neural-LLM Pipeline for Morphological Glossing in Endangered Language Documentation: A Case Study of Jungar Tuvancited by 0
  5. 2026Typological Feature Prediction with Large Language Models: An In-Context Learning Approachcited by 0
  6. 2026LLM Agents as Computational Typologistscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs
  2. 2025IOLBENCH: Benchmarking LLMs on Linguistic Reasoning
  3. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  4. 2026Reasoning over Grammar: Can Synthetic Linguistic Reasoning Traces Enhance Low-Resource Machine Translation?
  5. 2025UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?
  6. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilities
  7. 2025Cross-Lingual Consistency: A Novel Inference Framework for Advancing Reasoning in Large Language Models
  8. 2024LLMs' Understanding of Natural Language Revealed