mlsanityData to 14 Oct 2026

UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?

arXiv 2508.11260 ↗First posted 15 Aug 2025cs.CLcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have demonstrated potential in reasoning tasks, but their performance on linguistics puzzles remains consistently poor. These puzzles, often derived from Linguistics Olympiad (LO) contests, provide a minimal contamination environment to assess LLMs' linguistic reasoning abilities across low-resource languages. This work analyses LLMs' performance on 629 problems across 41 low-resource languages by labelling each with linguistically informed features to unveil weaknesses. Our analyses show that LLMs struggle with puzzles involving higher morphological complexity and perform better on puzzles involving linguistic features that are also found in English. We also show that splitting words into morphemes as a pre-processing step improves solvability, indicating a need for more informed and language-specific tokenisers. These findings thus offer insights into some challenges in linguistic reasoning and modelling of low-resource languages.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 27 of its 42 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

27 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025BIG-Bench Extra Hard26 Feb
  2. 2025Reasoning Over the Glyphs: Evaluation of LLM's Decipherment of Rare Scripts29 Jan
  3. 2024Inductive Linguistic Reasoning with Large Language Models9 Dec
  4. 2024Linguini: A benchmark for language-agnostic linguistic reasoning18 Sep
  5. 2024modeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models24 Jun
  6. 2024LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages10 Jun

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  5. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 195
  6. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  7. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  8. 2025Fanar: An Arabic-Centric Multimodal Generative AI Platformcited by 67
  9. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  10. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  11. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 25
  12. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 19
9 more
  1. 2020PuzzLing Machines: A Challenge on Learning From Small Datacited by 15
  2. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  3. 2024T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddingscited by 10
  4. 2024Tokenisation is NP-Completecited by 10
  5. 2023Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Modelcited by 9
  6. 2023Morphosyntactic probing of multilingual BERT modelscited by 8
  7. 2025Tokenization Standards for Linguistic Integrity: Turkish as a Benchmarkcited by 8
  8. 2019Extending Machine Language Models toward Human-Level Language Understandingcited by 7
  9. 2025A Partition Cover Approach to Tokenizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation Analysiscited by 4
  2. 2026Think Through a Bottleneck: Hourglass Reasoning for Rigorous Inductioncited by 0
  3. 2026Load-Bearing Context: The Question Damage Score for Evaluating Context Reliance in Linguistic Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
  2. 2026The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoning
  3. 2025IOLBENCH: Benchmarking LLMs on Linguistic Reasoning
  4. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilities
  5. 2025LingGym: How Far Are LLMs from Thinking Like Field Linguists?
  6. 2024Puzzle Solving using Reasoning of Large Language Models: A Survey
  7. 2025The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs using Indian Riddles
  8. 2025Reasoning Effort and Problem Complexity: A Scaling Analysis in LLMs