mlsanityData to 7 Oct 2026

Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discovery

arXiv 2502.15224 ↗First posted 21 Feb 2025cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Interactive discovery requires agents to maintain and update structured beliefs over many rounds of feedback. Before evaluating agents in noisy, open-ended scientific environments, it is useful to isolate this prerequisite capability under controlled conditions. We introduce Auto-Discovery-Bench, a deterministic oracle-guided diagnostic benchmark in which agents recover hidden structures through repeated hypothesis--intervention--feedback cycles. The benchmark instantiates three controlled discovery abstractions: directed graph discovery, undirected relational discovery, and symbolic equation discovery. Across models, performance degrades as the number of variables, trajectory length, and distractors increase. A separate trajectory-tracking diagnostic shows that many failures persist even when intervention selection and hypothesis generation are removed, suggesting that limitations in maintaining and integrating long-range structured information are an important bottleneck for oracle-guided discovery. Auto-Discovery-Bench is not intended to replace realistic discovery environments; rather, it provides a reproducible, low-confound diagnostic testbed for isolating a prerequisite capability for interactive scientific agents.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 11 of its 23 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

11 of 23 references.

References · 11

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  4. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  7. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  8. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 89
  9. 2023Can large language models build causal graphs?cited by 72
  10. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  11. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  2. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  3. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 32
  4. 2026Large Language Model Reasoning Failurescited by 20
  5. 2025The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvementscited by 12
  6. 2025CellForge: Agentic Design of Virtual Cell Modelscited by 11
  7. 2025A Survey of AI Scientistscited by 9
  8. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 9
  9. 2026Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world modelscited by 6
  10. 2026WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Modelscited by 4
  11. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  12. 2026CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientistscited by 2
3 more
  1. 2026Enhanced Polarization Locking in VCSELscited by 1
  2. 2026AlphaEval: Evaluating Agents in Productioncited by 1
  3. 2026GlowGS: Generative Semantic Feature Learning for 3D Gaussian Splatting in Nighttime Glow Scenescited by 1