# MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models

arXiv: https://arxiv.org/abs/2410.09542 · HTML: https://mlsanity.com/paper/2410.09542/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 12 Oct 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization](https://mlsanity.com/paper/2405.15071/) (2024)
- [Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries](https://mlsanity.com/paper/2406.12775/) (2024)
- [Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners](https://mlsanity.com/paper/2305.14825/) (2023)
- [Large Language Models Are Not Strong Abstract Reasoners](https://mlsanity.com/paper/2305.19555/) (2023)
- [MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning](https://mlsanity.com/paper/2404.13591/) (2024)
- [Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs](https://mlsanity.com/paper/2408.00114/) (2024)
- [Case2Code: Scalable Synthetic Data for Code Generation](https://mlsanity.com/paper/2407.12504/) (2024)
- [When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives](https://mlsanity.com/paper/2406.12084/) (2024)

## Built on by (most cited first, 17 of 17)

- [Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills](https://mlsanity.com/paper/2603.25158/) (2026)
- [A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2605.07358/) (2026)
- [CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis](https://mlsanity.com/paper/2503.23145/) (2025)
- [MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?](https://mlsanity.com/paper/2502.09933/) (2025)
- [From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Models](https://mlsanity.com/paper/2505.21935/) (2025)
- [Reasoning Can Hurt the Inductive Abilities of Large Language Models](https://mlsanity.com/paper/2505.24225/) (2025)
- [InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning](https://mlsanity.com/paper/2509.12263/) (2025)
- [Improving In-Context Learning with Reasoning Distillation](https://mlsanity.com/paper/2504.10647/) (2025)
- [Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interaction](https://mlsanity.com/paper/2508.19035/) (2025)
- [Deductive Logic in Language Models: Horizontal vs Vertical Reasoning](https://mlsanity.com/paper/2510.09340/) (2025)
- [HERO'S JOURNEY: Testing Complex Rule Induction with Text Games](https://mlsanity.com/paper/2606.02556/) (2026)
- [On the Role of Model Prior in Real-World Inductive Reasoning](https://mlsanity.com/paper/2412.13645/) (2024)
- [Visual Perceptual to Conceptual First-Order Rule Learning Networks](https://mlsanity.com/paper/2604.07897/) (2026)
- [Strategy-Induct: Task-Level Strategy Induction for Instruction Generation](https://mlsanity.com/paper/2605.20924/) (2026)
- [PraMem: Practice-derived Experiential Memory for Long-horizon Behavior Prediction](https://mlsanity.com/paper/2607.02881/) (2026)
- [Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Models](https://mlsanity.com/paper/2608.20975/) (2026)
- [SIMLIFE: Pattern Understanding for Long-Horizon Human-Agent Partnership](https://mlsanity.com/paper/2609.19610/) (2026)

## Abstract (arXiv)

Inductive reasoning is an essential capability for large language models (LLMs) to achieve higher intelligence, which requires the model to generalize rules from observed facts and then apply them to unseen examples. We present MIRAGE, a synthetic dataset that addresses the limitations of previous work, specifically the lack of comprehensive evaluation and flexible test data. In it, we evaluate LLMs' capabilities in both the inductive and deductive stages, allowing for flexible variation in input distribution, task scenario, and task difficulty to analyze the factors influencing LLMs' inductive reasoning. Based on these multi-faceted evaluations, we demonstrate that the LLM is a poor rule-based reasoner. In many cases, when conducting inductive reasoning, they do not rely on a correct rule to answer the unseen case. From the perspectives of different prompting methods, observation numbers, and task forms, models tend to consistently conduct correct deduction without correct inductive rules. Besides, we find that LLMs are good neighbor-based reasoners. In the inductive reasoning process, the model tends to focus on observed facts that are close to the current test example in feature space. By leveraging these similar examples, the model maintains strong inductive capabilities within a localized region, significantly improving its deductive performance.