# Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints

arXiv: https://arxiv.org/abs/2608.26197 · HTML: https://mlsanity.com/paper/2608.26197/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 4 of its 14 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 4 of 4)

- [Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agents](https://mlsanity.com/paper/2605.22166/) (2026)
- [How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines](https://mlsanity.com/paper/2605.28840/) (2026)
- [R-LAM: Reproducibility-Constrained Large Action Models for Scientific Workflow Automation](https://mlsanity.com/paper/2601.09749/) (2026)
- [Beyond State Machines: Executing Network Procedures with Agentic Tool-Calling Sequences](https://mlsanity.com/paper/2605.02584/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) based agents exhibit substantial run-to-run execution variance even when given identical tasks and tools -- acceptable for exploratory use but unacceptable in regulated domains such as finance and compliance. We study harness engineering: wrapping an agent in a deterministic execution layer (finite-state control, forced tool selection, output validation, bounded retry, and structured planning) and measuring its effect on execution determinism and task success. Across two synthetic tasks (finance and legal) and two open-weight models (Qwen-2.5-7B-Instruct, Gemma-3-27B), a first-pass harness produces a mixed result: it significantly improves reproducibility in one of four model-task cells, significantly degrades it in two, and has no effect in the fourth. A trace-level diagnostic finds the cause: once tool sequence, state sequence, and output are already highly consistent, an unconstrained free-text planning step becomes the dominant remaining source of variance. Adding Structured Planning -- validating the plan against a fixed schema before any tool is invoked -- eliminates the effect entirely: three of four cells reach a Reproducibility Rate and Determinism Index of 1.000 at N=100, and task success rises to 100% in three of four cells. The gain is not free: token cost falls in every cell, but latency shows a genuine, sample-size-robust split by model -- one becomes faster under the constraint, the other markedly slower. We argue harness engineering is a distinct, effective discipline for agent reliability, but its cost is model-dependent and must be measured, not assumed.