Scaling Laws for Agent Harnesses via Effective Feedback Compute
Not on the map. Why?
Abstract (arXiv)
Agent harnesses shape language-model performance by controlling tool use, feedback, verification, memory, and repair. Yet raw test-time expenditure, such as tokens, tool calls, wall time, or cost, cannot distinguish useful feedback from redundant or unstable interaction. We introduce Effective Feedback Compute (EFC), a trace-level scaling coordinate for informative, valid, non-redundant, and retained feedback. We further define Estimated-EFC, NRS-EFC, harness efficiency η, and task-demand normalization for realistic traces and heterogeneous tasks. Across synthetic, real, held-out, and prospective evaluations, EFC-based coordinates outperform raw-compute baselines and SAS. Oracle-EFC/Dtask reaches R2=0.99 in controlled scaling, and NRS-EFC/Dtask reaches R2=0.93 on real traces where raw compute has near-zero or negative fit. Finally, ours uses EFC as a companion control layer for existing harnesses, improving mean pass rate from 61.2% to 68.2% while reducing mean raw cost from 213.8 to 85.1 under matched settings. These results suggest that harness scaling depends on durable, task-sufficient feedback rather than raw computation alone.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 28 May 2026.
Builds on
24 of 31 references.
References · 24
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Let's Verify Step by Stepcited by 3,598
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
- 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
12 more
- 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
- 2025Towards a Science of Scaling Agent Systemscited by 125
- 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
- 2025Scaling Test-time Compute for LLM Agentscited by 59
- 2026Natural-Language Agent Harnessescited by 54
- 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 42
- 2026Benchmark Test-Time Scaling of General LLM Agentscited by 24
- 2026ContextBench: A Benchmark for Context Retrieval in Coding Agentscited by 23
- 2026Scaling Test-Time Compute for Agentic Codingcited by 11
- 2026Agentic Test-Time Scaling for WebAgentscited by 7
- 2026The Scaling Laws of Skills in LLM Agent Systemscited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.