# Inductive Linguistic Reasoning with Large Language Models

arXiv: https://arxiv.org/abs/2412.17819 · HTML: https://mlsanity.com/paper/2412.17819/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://mlsanity.com/paper/2405.15032/) (2024)
- [Large Language Models can Learn Rules](https://mlsanity.com/paper/2310.07064/) (2023)
- [Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator](https://mlsanity.com/paper/2206.08082/) (2022)
- [Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond](https://mlsanity.com/paper/2306.09841/) (2023)
- [Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners](https://mlsanity.com/paper/2305.14825/) (2023)
- [Large Language Models Are Not Strong Abstract Reasoners](https://mlsanity.com/paper/2305.19555/) (2023)
- [LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages](https://mlsanity.com/paper/2406.06196/) (2024)
- [Auto-ICL: In-Context Learning without Human Supervision](https://mlsanity.com/paper/2311.09263/) (2023)

## Built on by (most cited first, 2 of 2)

- [UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?](https://mlsanity.com/paper/2508.11260/) (2025)
- [Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages](https://mlsanity.com/paper/2602.02182/) (2026)

## Abstract (arXiv)

Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. In this work, we investigate the abilities of such models to perform abstract multilingual reasoning through the lens of linguistic puzzles on extremely low-resource languages. As these translation tasks involve inductive and deductive reasoning from reference instances, we examine whether diverse auxiliary demonstrations can be automatically induced from seed exemplars, through analogical prompting. We employ a two-stage procedure, first generating analogical exemplars with a language model, and then applying them in-context along with provided target language exemplars. Our results on the modeLing dataset show that analogical prompting is effective in eliciting models' knowledge of language grammar similarities, boosting the performance of GPT-4o by as much as 8.1% and Llama-3.1-405B-Instruct by 5.9% over chain-of-thought approaches. These gains are attributable to the analogical demonstrations, both when self-generated as well as when produced by weaker multilingual models. Furthermore, we demonstrate that our method generalizes to other tasks present in Linguistics Olympiad competitions, achieving sizable improvements across all problem types and difficulty levels included in the LINGOLY dataset with GPT-4o. We also report several findings about interesting phenomena which drive linguistic reasoning performance, suggesting that such puzzles are a valuable benchmark for new reasoning methods.