# HARBOR: Automated Harness Optimization arXiv: https://arxiv.org/abs/2604.20938 ยท HTML: https://mlsanity.com/paper/2604.20938/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 23 of its 42 references that are in the map. - First posted on arXiv 22 Apr 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 23 of 23) - [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023) - [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023) - [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016) - [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022) - [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024) - [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018) - [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019) - [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024) - [Meta-Harness: End-to-End Optimization of Model Harnesses](https://mlsanity.com/paper/2603.28052/) (2026) - [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022) - [High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces](https://mlsanity.com/paper/2103.00349/) (2021) - [Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning](https://mlsanity.com/paper/1903.04703/) (2019) - [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014) - [Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes](https://mlsanity.com/paper/1706.03673/) (2017) - [Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems](https://mlsanity.com/paper/2604.14228/) (2026) - [Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization](https://mlsanity.com/paper/2210.10199/) (2022) - [Difficulty-Aware Agentic Orchestration for Query-Specific Multi-Agent Workflows](https://mlsanity.com/paper/2509.11079/) (2025) - [Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving](https://mlsanity.com/paper/2603.18897/) (2026) - [Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation](https://mlsanity.com/paper/2508.07675/) (2025) - [Pancake: Hierarchical Memory System for Multi-Agent LLM Serving](https://mlsanity.com/paper/2602.21477/) (2026) - [Safe Risk-averse Bayesian Optimization for Controller Tuning](https://mlsanity.com/paper/2306.13479/) (2023) - [Agentic Code Optimization via Compiler-LLM Cooperation](https://mlsanity.com/paper/2604.04238/) (2026) - [From Translation to Superset: Benchmark-Driven Evolution of a Production AI Agent from Rust to Python](https://mlsanity.com/paper/2604.11518/) (2026) ## Built on by (most cited first, 11 of 11) - [AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces](https://mlsanity.com/paper/2608.23041/) (2026) - [Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses](https://mlsanity.com/paper/2606.02373/) (2026) - [Automated Discovery Has No Universally Superior Harness](https://mlsanity.com/paper/2607.18235/) (2026) - [Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems](https://mlsanity.com/paper/2609.00006/) (2026) - [Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence](https://mlsanity.com/paper/2608.21156/) (2026) - [Agentic-imodels: Evolving agentic interpretability tools via autoresearch](https://mlsanity.com/paper/2605.03808/) (2026) - [Competing at Every Price Point with Agentic Evolution over a Menu of LLMs](https://mlsanity.com/paper/2608.16207/) (2026) - [Mixture of Self-Improving Branches For Agent Harness Optimization](https://mlsanity.com/paper/2609.37834/) (2026) - [Turbo Harness: Instance-Adaptive Harness Optimization](https://mlsanity.com/paper/2609.40330/) (2026) - [ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization](https://mlsanity.com/paper/2610.00906/) (2026) - [SHarP: Saliency-based Pruning of Agent Harnesses](https://mlsanity.com/paper/2610.04178/) (2026) ## Abstract (arXiv) Long-horizon language-model agents are dominated, in lines of code and in operational complexity, not by their underlying model but by the harness that wraps it: context compaction, tool caching, semantic memory, trajectory reuse, speculative tool prediction, and the glue that binds the model to a sandboxed execution environment. We argue that harness design is a first-class machine-learning problem and that automated configuration search dominates manual stacking once the flag space exceeds a handful of bits. We defend this claim in two steps. First, we formalize automated harness optimization as constrained noisy Bayesian optimization over a mixed-variable, cost-heterogeneous configuration space with cold-start-corrected rewards and a posterior chance-constrained safety check, and give a reference solver, HARBOR (Harness Axis-aligned Regularized Bayesian Optimization Routine), built from a block-additive SAAS surrogate, multi-fidelity cost-aware acquisition, and TuRBO trust regions. Second, we instantiate the problem in a flag-gated harness over a production coding agent and report a controlled four-round manual-tuning case study against a fixed task suite and an end-to-end HARBOR run. The formulation itself is task-class agnostic: the configuration space, reward correction, acquisition, and safety check apply to any agent harness with a bounded flag space and a reproducible task suite.