# LingGym: How Far Are LLMs from Thinking Like Field Linguists?

arXiv: https://arxiv.org/abs/2511.00343 · HTML: https://mlsanity.com/paper/2511.00343/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 29 of its 54 references that are in the map.
- First posted on arXiv 1 Nov 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 29 of 29)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [What do RNN Language Models Learn about Filler-Gap Dependencies?](https://mlsanity.com/paper/1809.00042/) (2018)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [A Benchmark for Learning to Translate a New Language from One Grammar Book](https://mlsanity.com/paper/2309.16575/) (2023)
- [ByT5 model for massively multilingual grapheme-to-phoneme conversion](https://mlsanity.com/paper/2204.03067/) (2022)
- [The Zero Resource Speech Challenge 2017](https://mlsanity.com/paper/1712.04313/) (2017)
- [Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages](https://mlsanity.com/paper/2404.11553/) (2024)
- [Teaching Large Language Models an Unseen Language on the Fly](https://mlsanity.com/paper/2402.19167/) (2024)
- [LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages](https://mlsanity.com/paper/2406.06196/) (2024)
- [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025)
- [Can we teach language models to gloss endangered languages?](https://mlsanity.com/paper/2406.18895/) (2024)
- [Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning](https://mlsanity.com/paper/2411.18571/) (2024)
- [Can LLMs Help Create Grammar?: Automating Grammar Creation for Endangered Languages with In-Context Learning](https://mlsanity.com/paper/2412.10960/) (2024)
- [Multiple Sources are Better Than One: Incorporating External Knowledge in Low-Resource Glossing](https://mlsanity.com/paper/2406.11085/) (2024)
- [GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning](https://mlsanity.com/paper/2410.18702/) (2024)
- [Developing multilingual speech synthesis system for Ojibwe, Mi'kmaq, and Maliseet](https://mlsanity.com/paper/2502.02703/) (2025)

## Built on by (most cited first, 6 of 6)

- [Massively Multilingual Joint Segmentation and Glossing](https://mlsanity.com/paper/2601.10925/) (2026)
- [Speaking of Language: Reflections on Metalanguage Research in NLP](https://mlsanity.com/paper/2604.02645/) (2026)
- [Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages](https://mlsanity.com/paper/2602.02182/) (2026)
- [Hybrid Neural-LLM Pipeline for Morphological Glossing in Endangered Language Documentation: A Case Study of Jungar Tuvan](https://mlsanity.com/paper/2603.00923/) (2026)
- [Typological Feature Prediction with Large Language Models: An In-Context Learning Approach](https://mlsanity.com/paper/2609.03775/) (2026)
- [LLM Agents as Computational Typologists](https://mlsanity.com/paper/2609.07791/) (2026)

## Abstract (arXiv)

This paper introduces LingGym, a new benchmark that evaluates LLMs' capacity for meta-linguistic reasoning using Interlinear Glossed Text (IGT) and grammatical descriptions extracted from 18 typologically diverse reference grammars. Unlike previous work that focuses on specific downstream tasks, we assess whether LLMs can generalize linguistic inference across low-resource languages and structures not seen during training. We present a controlled evaluation task: Word-Gloss Inference, in which the model must infer a missing word and gloss from context using varying levels of linguistic information (e.g., glosses, grammatical explanations, translations). Our results show that incorporating structured linguistic cues leads to consistent improvements in reasoning performance across all models. This work highlights both the promise and current limitations of using LLMs for typologically informed linguistic analysis and low-resource language documentation.