# FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games

arXiv: https://arxiv.org/abs/2606.04751 · HTML: https://mlsanity.com/paper/2606.04751/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 8 of 8)

- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [WILT: A Multi-Turn, Memorization-Robust Inductive Logic Benchmark for LLMs](https://mlsanity.com/paper/2410.10998/) (2024)
- [Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models](https://mlsanity.com/paper/2604.02485/) (2026)

## Built on by (most cited first, 1 of 1)

- [DiG-bench: Discovery in Games](https://mlsanity.com/paper/2608.12593/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed as autonomous agents in scientific tasks. Yet whether these systems can effectively engage in forms of inductive reasoning relevant to scientific discovery remains an open question. In this work, we introduce FALSIFYBENCH, an evaluation framework for hypothesis-driven reasoning inspired by the classic Wason 2-4-6 task, in which agents must discover hidden semantic properties by iteratively proposing examples and receiving feedback. This task captures key elements of scientific reasoning: hypothesis generation, evidence gathering, and belief revision in response to both confirming and disconfirming evidence. Our evaluation of 12 LLMs across model families and scales shows that reasoning models are generally stronger scientific reasoners than instruction-tuned models, although no model comes close to optimal performance. The primary driver of success is the capacity for negative testing: models that actively seek to falsify their hypotheses consistently outperform those that primarily seek confirmation. Moreover, a fine-grained turn-level analysis, neglected in previous work, reveals that failure is tied to identifiable patterns in how models navigate the hypothesis space.