mlsanityData to 7 Oct 2026

Search Discipline for Long-Horizon Research Agents

arXiv 2606.11522 ↗First posted 9 Jun 2026cs.AI cs.LGcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Autoresearch agents now propose, evaluate, and select scientific candidates against a metric, and that metric is usually an aggregate reduced over a heterogeneous space of regions, slices, or cohorts. We show that when scientific validity lives in that disaggregated structure, the aggregate can rank the wrong candidate first. The headline number improves while the structure underneath inverts, so a decision made on the number accepts a candidate that quietly breaks the model. The failure is not domain-specific. It appears wherever a candidate's validity is multi-dimensional but its verifier is a single reduction. We demonstrate the inversion on a fire-model task in the Ecosystem Demography model. The highest-scoring candidate and a slightly lower one are within noise of each other on global score, yet the top-scoring one collapses the protected boreal regions while the other preserves them. What separates them is the per-region behavior, not the headline number. This decision should not be left to the agent that produced the candidates. The agent optimizing the score is the last party likely to catch the score being wrong, and a prompt has no remaining turn once the agent has stopped. We move the decision to an external control loop that audits each candidate on its disaggregated behavior and acts after the agent has decided. It can demote a candidate the agent would have accepted, and it can reopen a run the agent had declared finished. Our contribution is the inversion finding itself, and a search-discipline protocol that decides on reviewable candidate-effect evidence instead of the score.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 9 of its 14 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

9 of 14 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search10 Apr
  2. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering9 Oct

Other references · 7

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  6. 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 215
  7. 2026Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Softwarecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Recovering Wasted Compute in Autoresearch Agents
  2. 2026The Greatness of Science Cannot Be Planned: Agentic Auto-Research is Fuzz Testing
  3. 2026LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
  4. 2026Scaling Automatic Research Agents via World Models
  5. 2026ST-Bench: A Spatial-Temporal Benchmark for Multi-Agent System Generation on Scientific Research Tasks
  6. 2026ArcticSwarm: Deferring Early Consensus in Long-Horizon Multi-Agent Research
  7. 2026AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration
  8. 2026How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks