# More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses

arXiv: https://arxiv.org/abs/2609.35873 · HTML: https://mlsanity.com/paper/2609.35873/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [More Agents Is All You Need](https://mlsanity.com/paper/2402.05120/) (2024)
- [Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses](https://mlsanity.com/paper/2604.25850/) (2026)
- [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://mlsanity.com/paper/2308.02828/) (2023)
- [Self-Harness: Harnesses That Improve Themselves](https://mlsanity.com/paper/2606.09498/) (2026)
- [LLM-as-a-Verifier: A General-Purpose Verification Framework](https://mlsanity.com/paper/2607.05391/) (2026)
- [From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws](https://mlsanity.com/paper/2606.06324/) (2026)
- [HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems](https://mlsanity.com/paper/2606.01779/) (2026)
- [TTHE: Test-Time Harness Evolution](https://mlsanity.com/paper/2607.08124/) (2026)
- [JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution](https://mlsanity.com/paper/2608.25593/) (2026)
- [HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution](https://mlsanity.com/paper/2607.13683/) (2026)
- [Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable](https://mlsanity.com/paper/2607.13285/) (2026)
- [GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents](https://mlsanity.com/paper/2605.29668/) (2026)
- [HELIX: Model-Harness Co-evolution for Recursive Self-Improvement](https://mlsanity.com/paper/2608.13951/) (2026)
- [Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification](https://mlsanity.com/paper/2608.27311/) (2026)
- [Automated Discovery Has No Universally Superior Harness](https://mlsanity.com/paper/2607.18235/) (2026)
- [Inefficiencies of Meta Agents for Agent Design](https://mlsanity.com/paper/2510.06711/) (2025)
- [Select-then-Solve: Paradigm Routing as Inference-Time Optimization for LLM Agents](https://mlsanity.com/paper/2604.06753/) (2026)
- [Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection](https://mlsanity.com/paper/2608.20169/) (2026)
- [When Should LLMs Search? Counterfactual Supervision for Search Routing](https://mlsanity.com/paper/2607.05752/) (2026)
- [It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers](https://mlsanity.com/paper/2605.26731/) (2026)

## Built on by (most cited first, 1 of 1)

- [Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation](https://mlsanity.com/paper/2610.10368/) (2026)

## Abstract (arXiv)

Automated generation of LLM harnesses promises to improve inference through task specialization. Yet additional answer coverage can arise from repeated execution of the same program, making specialization difficult to identify. We introduce a controlled evaluation that separates answer coverage, repeatable task advantages, and gains from pre-execution selection. On 386 MATH-500 tasks, we compare eight generated harnesses plus a baseline with nine byte-identical baseline copies, using three executions per member. Identical programs yield 2.16 percentage points of repeat-averaged oracle headroom. Generated programs exhibit substantially more repeatable score patterns, but these chiefly reveal persistent weaknesses: losses relative to the baseline persist across all three repeats on 100 tasks, while persistent wins occur on only one task and are sensitive to answer extraction. The frozen selector gains 0.00 percentage points, and both populations reach 98.70% oracle coverage at 27 harness executions. Stable complementarity remains unresolved at three repeats. Supporting BIRD traces locate failures in mechanism implementation, activation, and output validity. Together, these findings establish why coverage and repeatability alone cannot justify claims of useful specialization. They motivate an evaluation standard for harness diversity: task advantages should persist across executions, guide usable decisions, and improve on additional fixed-program executions under matched inference budgets.