# GEAR: A General Evaluation Framework for Abductive Reasoning

arXiv: https://arxiv.org/abs/2509.24096 · HTML: https://mlsanity.com/paper/2509.24096/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 12 of its 58 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 12 of 12)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas](https://mlsanity.com/paper/2410.14255/) (2024)
- [Logical Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2502.09100/) (2025)
- [Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges](https://mlsanity.com/paper/2409.02387/) (2024)
- [Logical Reasoning over Natural Language as Knowledge Representation: A Survey](https://mlsanity.com/paper/2303.12023/) (2023)
- [Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining](https://mlsanity.com/paper/2502.06733/) (2025)
- [InductionBench: LLMs Fail in the Simplest Complexity Class](https://mlsanity.com/paper/2502.15823/) (2025)

## Built on by (most cited first, 6 of 6)

- [Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMs](https://mlsanity.com/paper/2604.08016/) (2026)
- [AILS-NTUA at SemEval-2026 Task 12: Graph-Based Retrieval and Reflective Prompting for Abductive Event Reasoning](https://mlsanity.com/paper/2603.04319/) (2026)
- [Explanation Quality Assessment as Ranking with Listwise Rewards](https://mlsanity.com/paper/2604.24176/) (2026)
- [LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos](https://mlsanity.com/paper/2607.12733/) (2026)
- [CEDAR-GRPO: Process-Aware Reinforcement Learning for General Abductive Reasoning in LLMs](https://mlsanity.com/paper/2608.14791/) (2026)
- [When the Canonical Completion Is Wrong: Formalizing and Measuring the Jump in Large Language Models](https://mlsanity.com/paper/2608.26187/) (2026)

## Abstract (arXiv)

Since the advent of large language models (LLMs), research has focused on instruction following and deductive reasoning. A central question remains: can these models discover new knowledge, and how can we evaluate this ability? We address this by studying abductive reasoning-the generation of plausible hypotheses to explain observations-and introduce GEAR (General Evaluation for Abductive Reasoning), a general-purpose, fully automated, transparent, and label-free evaluation paradigm. GEAR scores hypothesis sets by three metrics: consistency (each hypothesis explains the observations), generalizability (consistent hypotheses make meaningful predictions on unseen inputs), and diversity (the set covers distinct predictions and patterns). Built this way, GEAR is scalable (no human gold answers), reliable (deterministic scoring aligned with classical abduction), and open-ended (scores improve only when models produce new plausible hypotheses, unlike static benchmarks that saturate once accuracy is high). Using GEAR, we conduct a fine-grained study of nine LLMs on four abduction benchmarks with 1,500 problems, generating over 50,000 candidate hypotheses and revealing model differences obscured by gold-answer or purely human evaluations. We further propose a momentum-based curriculum that adjusts GEAR-derived training data by learning velocity: it starts with what the model learns quickly and shifts toward harder objectives such as generating diverse hypotheses once the model is confident on foundational objectives. Without gold-label supervision, this strategy improves all GEAR objectives and these gains transfer to established abductive reasoning benchmarks. Taken together, GEAR provides a principled framework that evaluates abduction and supplies label-free, scalable training signals that help LLMs produce more diverse and reliable hypotheses.