# What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents

arXiv: https://arxiv.org/abs/2609.32318 · HTML: https://mlsanity.com/paper/2609.32318/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 29 of 29)

- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019)
- [Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.00212/) (2025)
- [Optimal Transport for structured data with application on graphs](https://mlsanity.com/paper/1805.09114/) (2018)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [Scalable methods for computing state similarity in deterministic Markov Decision Processes](https://mlsanity.com/paper/1911.09291/) (2019)
- [AutoSurvey: Large Language Models Can Automatically Write Surveys](https://mlsanity.com/paper/2406.10252/) (2024)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Causal Consistency of Structural Equation Models](https://mlsanity.com/paper/1707.00819/) (2017)
- [DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research](https://mlsanity.com/paper/2511.19399/) (2025)
- [The Leaderboard Illusion](https://mlsanity.com/paper/2504.20879/) (2025)
- [SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writing](https://mlsanity.com/paper/2503.04629/) (2025)
- [Compositional Abstraction Error and a Category of Causal Models](https://mlsanity.com/paper/2103.15758/) (2021)
- [Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewards](https://mlsanity.com/paper/2507.03041/) (2025)
- [SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation](https://mlsanity.com/paper/2508.15658/) (2025)
- [Causal Optimal Transport of Abstractions](https://mlsanity.com/paper/2312.08107/) (2023)
- [LiRA: A Multi-Agent Framework for Reliable and Readable Literature Review Generation](https://mlsanity.com/paper/2510.05138/) (2025)
- [SurveyG: A Multi-Agent LLM Framework with Hierarchical Citation Graph for Automated Survey Generation](https://mlsanity.com/paper/2510.07733/) (2025)
- [Log analysis is necessary for credible evaluation of AI agents](https://mlsanity.com/paper/2605.08545/) (2026)
- [SurveyGen-I: Consistent Scientific Survey Generation with Evolving Plans and Memory-Guided Writing](https://mlsanity.com/paper/2508.14317/) (2025)
- [KV-Skill: Forging Expertise in the Model's Native Language](https://mlsanity.com/paper/2608.05475/) (2026)
- ["LLM Agent Performance" Is Not a Single Evaluation Target](https://mlsanity.com/paper/2602.03238/) (2026)
- [ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents](https://mlsanity.com/paper/2512.00986/) (2025)
- [OTAP: Structure-Aware Optimal Transport for Evaluating Planning and Execution in Agent Trajectories](https://mlsanity.com/paper/2607.17082/) (2026)

## Abstract (arXiv)

Leaderboards rank long-horizon agents by their final outputs. Yet a higher score alone does not establish whether two systems are comparable or which stage accounts for the difference. Unequal evidence, inputs, or budgets can affect scores, and statistical corrections do not remove this mismatch. We introduce compositional controllability to address these questions. A comparison window covers one stage, several stages, or the whole agent. Our central result bounds the gap between observed and controlled score differences using only nuisance outside the window. This yields an admissibility test applied before scores are inspected. Inadmissible comparisons are refused. For admissible pairs, an ordering is certified only when the score gap exceeds the combined sampling and nuisance radii; otherwise, it remains undecided. These decisions give each system a rank interval. We introduce BioLitBench, a benchmark of 2,042 biomedical articles represented as structured claim graphs. Among seven published pipelines, a conventional statistical analysis declares a winner in 14 of 21 pairwise comparisons. Yet the top-ranked system alone received the target review's bibliography. To isolate pipeline performance, our test requires matched inputs and a fixed backbone model. It refuses 11 of the 21 comparisons, including every comparison involving the top-ranked system. Seven of the 14 conventional conclusions fall within these refused pairs. The same comparison windows support stage-level training. We train SCRIBE on Qwen3.8-27B using rewards measured at each stage's exit. Under matched evidence, SCRIBE achieves a certified rank interval of [1,2], with certified advantages over all evaluated published pipelines and the evaluated Claude and OpenAI agents. Under same pool, SCRIBE matches the strongest published retriever and is certified above three published pipelines.