# SpecPath: Testing Coding Agents Across Contract-Equivalent Specification Histories

arXiv: https://arxiv.org/abs/2608.09799 · HTML: https://mlsanity.com/paper/2608.09799/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 25 of 25)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study](https://mlsanity.com/paper/2503.15223/) (2025)
- [Reverse-Engineered Reasoning for Open-Ended Generation](https://mlsanity.com/paper/2509.06160/) (2025)
- [To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization](https://mlsanity.com/paper/2502.00691/) (2025)
- [LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History](https://mlsanity.com/paper/2402.18216/) (2024)
- [Can Language Models Follow Multiple Turns of Entangled Instructions?](https://mlsanity.com/paper/2503.13222/) (2025)
- [MaintainCoder: Maintainable Code Generation Under Dynamic Requirements](https://mlsanity.com/paper/2503.24260/) (2025)
- [RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time](https://mlsanity.com/paper/2604.11626/) (2026)
- [SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions](https://mlsanity.com/paper/2606.30573/) (2026)
- [EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions](https://mlsanity.com/paper/2605.24110/) (2026)
- [Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systems](https://mlsanity.com/paper/2605.26302/) (2026)
- [From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions](https://mlsanity.com/paper/2502.13791/) (2025)
- [Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation](https://mlsanity.com/paper/2607.05382/) (2026)
- [Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning](https://mlsanity.com/paper/2605.14054/) (2026)
- [SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades](https://mlsanity.com/paper/2605.14415/) (2026)
- [Validating LLM-Generated Programs with Metamorphic Prompt Testing](https://mlsanity.com/paper/2406.06864/) (2024)
- [When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents](https://mlsanity.com/paper/2603.17104/) (2026)
- [Revisable by Design: A Theory of Streaming LLM Agent Execution](https://mlsanity.com/paper/2604.23283/) (2026)
- [Benchmarking Correctness and Security in Multi-Turn Code Generation](https://mlsanity.com/paper/2510.13859/) (2025)
- [RIFT: Reordered Instruction Following Testbed To Evaluate Instruction Following in Singular Multistep Prompt Structures](https://mlsanity.com/paper/2601.18924/) (2026)
- [Regression Accumulation in Multi-Turn LLM Programming Conversations](https://mlsanity.com/paper/2607.01855/) (2026)

## Built on by (most cited first, 1 of 1)

- [You Changed Your Mind, The Model Didn't: Demystifying Intent in Multi-Turn Dialogue](https://mlsanity.com/paper/2610.06496/) (2026)

## Abstract (arXiv)

Modern coding agents increasingly appear capable of following complex software requirements, yet their success leaves a critical ambiguity: do they resolve the active specification, or merely follow the most salient path by which it was stated? We identify specification-path sensitivity, a failure mode in which requirement histories that are equivalent in their final meaning lead the same agent system to produce behaviorally different programs. This reframes evolving-requirement evaluation as active-contract resolution: before writing code, an agent must determine which requirements still count. Building on this view, we introduce SpecPath, a diagnostic evaluation that holds the repository, final contract, verifier, agent system, and execution budget fixed while changing only the revision path that leads to the contract. Rather than treating each patch as an isolated pass or failure, SpecPath uses paired executable outcomes to reveal whether an agent realizes the same tested behavior across contract-equivalent histories. Across five calibrated software tasks and fourteen coding-agent configurations, aggregate direct and revision-history accuracy is nearly unchanged; nevertheless, 35 of 100 complete blocks that succeed on the direct specification fail on at least one equivalent history. These results show that implementation success on a consolidated request does not guarantee specification-path invariance. Evaluating evolving requirements therefore calls for controlled tests of whether agents are robust to the path by which a specification becomes final.