Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Agentic video-generation systems close a loop between a generator and a verifier: an LLM plans shots, calls a text-to-video model, and a multimodal judge decides whether the result satisfies the request. To diagnose where a long workflow fails, recent harnesses deliberately show the judge more than the video-the agent's execution trace, its plan, the narration it synthesized. We ask whether this auxiliary text moves the judge's verdict on purely visual requirements, holding the frames fixed. On a benchmark of 109 generated two-event clips with manual labels, in which the requested event is either visibly completed or visibly missing, a trace that reports a successful tool call makes three open-weight Qwen-VL judges (7B, 8B, 32B) accept 78--90% of the failures, up from 7--19% without text, and a contradicting trace makes them reject up to 100% of correct clips; an instruction to ``use only the frames'' does not remove the effect. Frontier closed judges are essentially unmoved on the same clips, showing that the vulnerability is a property of the judge's learned trust in tool logs rather than of the task. Plan-derived text carries no clip-specific information, so it can only shift a judge's operating point, and in a repair loop that shift becomes a cap on the true pass rate that no repair policy can exceed; the cap matches simulation to two decimals. In the loop, contamination is exploited without any adversarial agent: an honest LLM planner that always regenerates ends with a judge pass rate of 1.00 and a human-labelled pass rate of 0.28, and a pipeline in which a cheap checker writes its verdict into the trace launders that checker's errors into a stronger final judge (0.69 false accepts).
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 28 references that are in the map. First posted on arXiv 23 Sep 2026.
Builds on
23 of 28 references.
References · 23
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
- 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
- 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
- 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generationcited by 31
- 2025VISTA: A Test-Time Self-Improving Video Generation Agentcited by 26
- 2024GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaborationcited by 15
- 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 12
- 2026ViMax: Agentic Video Generationcited by 11
- 2026SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agentscited by 5
11 more
- 2026VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generationcited by 5
- 2026VideoAgent: All-in-One Framework for Video Understanding and Editingcited by 5
- 2025SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Modelscited by 3
- 2026When Vision-Language Models Judge Without Seeing: Exposing Informativeness Biascited by 3
- 2026On the Adversarial Robustness of Multimodal LLM Judgescited by 3
- 2026Evidence-RL: Towards Evidence-intensive Visual Reasoningcited by 3
- 2026OmniHalluc-L: Counterfactual Benchmarking and Modality-Perturbation Reliability Calibration for Long-Form Omni Hallucinationcited by 2
- 2026When Does a Video-Language Model Stop Watching? Reward Strength Controls the Formation and Reversal of Visual Shortcuts in Multimodal RLVRcited by 2
- 2026Better Call CineCrew: Consistent Ultra-Long Narrative-to-Film Generationcited by 2
- 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Modelscited by 1
- 2026EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidencecited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
- 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents
- 2026Parsing the Stream: A Live Trace Model for Long-Horizon Agents and Their Observers
- 2026Execution-First Synthetic Tool-Use Trace Generation for LLM Agents
- 2026TraceSIR: A Multi-Agent Framework for Structured Analysis and Reporting of Agentic Execution Traces
- 2026TraceLab: Characterizing Coding Agent Workloads for LLM Serving
- 2026Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
- 2025TRAIL: Trace Reasoning and Agentic Issue Localization