Learning Correct Behavior from Examples: Validating Sequential Execution in Autonomous Agents
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
As autonomous agents become increasingly sophisticated, validating their sequential behavior presents a significant challenge. Traditional testing approaches require manual specification, exact sequence matching, or thousands of training examples. We present a novel algorithm that automatically learns correct behavior from just 2-10 passing execution traces and validates new executions against this learned model. Our approach combines dominator analysis from compiler theory with multimodal large language model-powered semantic understanding to identify essential states and handle non-deterministic behavior. The system constructs a generalized ground truth model using Prefix Tree Acceptors, merges traces through multi-tiered equivalence detection, and validates new executions via topological subsequence matching. In controlled experiments, our system achieved high accuracy in detecting product bugs and false successes using only 3 training traces. This approach provides explainable validation results with coverage metrics and works across diverse domains including UI testing, code generation, and robotic processes.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 9 of its 32 references that are in the map. First posted on arXiv 4 May 2026.
Builds on
9 of 32 references.
Closest ancestors · 3
Same direction, newest first
- 2026Willful Disobedience: Automatically Detecting Failures in Agentic Traces25 Mar
- 2026ContextCov: Deriving and Enforcing Executable Constraints from Agent Instruction Files28 Feb
- 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectories2 Feb
Other references · 6
- 2014Explaining and Harnessing Adversarial Examplescited by 7,336
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,274
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,992
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,218
- 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Usecited by 62
- 2021Using Machine Learning to Generate Test Oracles: A Systematic Literature Reviewcited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025An Approach to Checking Correctness for Agentic Systems
- 2026AutoRefine: Compiling Trajectories into Validated Typed Agent Artifacts
- 2026TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces
- 2026TriCEGAR: A Trace-Driven Abstraction Mechanism for Agentic AI
- 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
- 2026Your Agent Has a Genome: Sequence-Level Behavioral Analysis and Runtime Governance of LLM-Powered Autonomous Agents
- 2026PBT-Bench: Benchmarking AI Agents on Property-Based Testing
- 2026Agentic Fuzzing: Opportunities and Challenges