# Effective Harness Engineering for Algorithm Discovery with Coding Agents

arXiv: https://arxiv.org/abs/2605.15221 · HTML: https://mlsanity.com/paper/2605.15221/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 13 May 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 13 of 13)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution](https://mlsanity.com/paper/2402.01145/) (2024)
- [Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model](https://mlsanity.com/paper/2401.02051/) (2024)
- [From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2505.13259/) (2025)
- [ThetaEvolve: Test-time Learning on Open Problems](https://mlsanity.com/paper/2511.23473/) (2025)
- [Algorithm Evolution Using Large Language Model](https://mlsanity.com/paper/2311.15249/) (2023)
- [CodeEvolve: an open source evolutionary coding agent for algorithmic discovery and optimization](https://mlsanity.com/paper/2510.14150/) (2025)
- [Scientific Algorithm Discovery by Augmenting AlphaEvolve with Deep Research](https://mlsanity.com/paper/2510.06056/) (2025)
- [Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learned](https://mlsanity.com/paper/2603.05344/) (2026)
- [AlphaResearch: Accelerating New Algorithm Discovery with Language Models](https://mlsanity.com/paper/2511.08522/) (2025)
- [Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning](https://mlsanity.com/paper/2410.15639/) (2024)

## Built on by (most cited first, 6 of 6)

- [Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops](https://mlsanity.com/paper/2607.07663/) (2026)
- [Automated Discovery Has No Universally Superior Harness](https://mlsanity.com/paper/2607.18235/) (2026)
- [Loreley: Repository-Scale Program Evolution with Quality-Diversity Search](https://mlsanity.com/paper/2608.19703/) (2026)
- [Evolution or Illusion? Rethinking Evaluation in LLM Evolutionary Search](https://mlsanity.com/paper/2609.19799/) (2026)
- [Causal Improvement Graph for Agentic Harness Optimization](https://mlsanity.com/paper/2610.05039/) (2026)
- [AgentDiscover: Autonomous Discovery with Minimal Search Scaffolding](https://mlsanity.com/paper/2610.05334/) (2026)

## Abstract (arXiv)

AlphaEvolve and FunSearch have demonstrated the potential of combining large language models (LLMs) with evolutionary search for automated algorithm discovery. However, discovery success is shaped not only by model capability but also significantly by the design of the execution infrastructure, i.e., the harness. This paper investigates effective harness design through three questions: under a fixed token budget, is it better to produce many algorithms with brief thought or fewer algorithms with deeper thought? How should the harness handle evaluation hacks, where generated programs exploit the scoring function? And how can agents that require full filesystem access execute safely in parallel? Using Vesper, an algorithm discovery framework that incorporates harness improvements addressing these questions, we evaluate on Circle Packing under the same token budget. Interestingly, generating fewer algorithms while thinking more deeply about each one achieved higher scores. That is, scaling the quality of each individual is more budget-efficient than scaling the number of evolutionary generations. Surprisingly, more capable models produced evaluation hacks at higher rates, making hack detection increasingly necessary as models scale.