# CruxBench: A Benchmark of Information Discovery

arXiv: https://arxiv.org/abs/2609.35879 · HTML: https://mlsanity.com/paper/2609.35879/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 59 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 18 of 18)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities](https://mlsanity.com/paper/2409.19839/) (2024)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction](https://mlsanity.com/paper/2508.11987/) (2025)
- [CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models](https://mlsanity.com/paper/2405.12063/) (2024)
- [LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena](https://mlsanity.com/paper/2510.17638/) (2025)
- [Pitfalls in Evaluating Language Model Forecasters](https://mlsanity.com/paper/2506.00723/) (2025)
- [Outcome-based Reinforcement Learning to Predict the Future](https://mlsanity.com/paper/2505.17989/) (2025)
- [Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain](https://mlsanity.com/paper/2406.17453/) (2024)
- [Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle](https://mlsanity.com/paper/2411.08324/) (2024)
- [LLMs Can Teach Themselves to Better Predict the Future](https://mlsanity.com/paper/2502.05253/) (2025)
- [Value of Information: A Framework for Human-Agent Communication](https://mlsanity.com/paper/2601.06407/) (2026)
- [Future-as-Label: Scalable Supervision from Real-World Outcomes](https://mlsanity.com/paper/2601.06336/) (2026)
- [AIA Forecaster: Technical Report](https://mlsanity.com/paper/2511.07678/) (2025)
- [Automating Forecasting Question Generation and Resolution for AI Evaluation](https://mlsanity.com/paper/2601.22444/) (2026)
- [Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting](https://mlsanity.com/paper/2605.27668/) (2026)

## Abstract (arXiv)

Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many complex real-world tasks is identifying which questions are worth asking in the first place: decomposing a difficult problem into subquestions -- which we call cruxes -- whose answers provide key steps on the path toward solving the target problem. To evaluate this capability of information discovery, we introduce CruxBench, a benchmark that grades LLM-generated questions by their Value of Information (VOI): how much a model-proposed crux updates beliefs about a target forecasting question. CruxBench enjoys a rare combination of three key properties: it is (1) contamination-resistant by construction, since ground truth is generated by future world events; (2) open-ended, admitting unbounded and complex text-based submissions rather than one correct numeric answer; and (3) grounded, with informativeness measured against quantified changes in real-world beliefs. We evaluate a diverse set of eight models on 293 target forecasting questions and find that VOI correlates highly with independent measures of model capability (r=0.90) and captures cruxes' usefulness for answering target questions. However, information discovery remains challenging even for frontier LLMs, which only narrowly outperform a random-timing baseline.