# Chronicle: Cut-Point Replay for Regression Testing of LLM Agents

arXiv: https://arxiv.org/abs/2609.20625 · HTML: https://mlsanity.com/paper/2609.20625/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 13 of 13)

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents](https://mlsanity.com/paper/2406.13352/) (2024)
- [TRAIL: Trace Reasoning and Agentic Issue Localization](https://mlsanity.com/paper/2505.08638/) (2025)
- [Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias](https://mlsanity.com/paper/2606.19544/) (2026)
- [DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems](https://mlsanity.com/paper/2512.06749/) (2025)
- [Get Experience from Practice: LLM Agents with Record & Replay](https://mlsanity.com/paper/2505.17716/) (2025)
- [Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures](https://mlsanity.com/paper/2606.08275/) (2026)
- [REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces](https://mlsanity.com/paper/2606.09071/) (2026)
- [Are Coding Agents Generating Over-Mocked Tests? An Empirical Study](https://mlsanity.com/paper/2602.00409/) (2026)
- [Automated structural testing of LLM-based agents: methods, framework, and case studies](https://mlsanity.com/paper/2601.18827/) (2026)
- [Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript](https://mlsanity.com/paper/2501.12680/) (2025)
- [Knowledge-Based Zero-Replay Debugging of Multi-Agent LLM Traces](https://mlsanity.com/paper/2606.14805/) (2026)

## Abstract (arXiv)

Large language model responses are non-deterministic, so failures in LLM agents are hard to reproduce: a failure depends on inference that is not bitwise reproducible, on tools that read changing state, and on a multi-step trajectory that a re-run rarely repeats. Record-and-replay makes a run reproducible, but existing agent tooling records runs only to trace or score them, not to test a code change against them. We present Chronicle, which records an agent run at its non-deterministic boundaries as immutable envelopes and replays it from the record. Its central operation, cut-point replay, serves a chosen subset of boundaries from the record and executes the complementary subset live with new code, turning a recorded incident into a regression test that runs in continuous integration. On a benchmark of 6 recorded failures with simulated model boundaries, recording adds 23 μs per crossing (0.008% of an assumed 300 ms model call), full replay issues zero model calls and is bit-stable across 20 repetitions, and cut-point tests fail on faulty code and pass on guarded and benign changes for all 6 incidents. In a mutation study of the guarded tools, cut-point tests catch every mutant that lets the recorded unsafe action through, while a baseline that stubs every boundary, using the same assertion, catches none. Chronicle and the benchmark are publicly available at https://github.com/theagentplane/chronicle.