# Accuracy and Cost Claims Do Not Survive Re-Execution in Agentic VideoQA

arXiv: https://arxiv.org/abs/2607.01469 · HTML: https://mlsanity.com/paper/2607.01469/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 20 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 16 of 16)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024)
- [Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning](https://mlsanity.com/paper/2508.04416/) (2025)
- [MoReVQA: Exploring Modular Reasoning Models for Video Question Answering](https://mlsanity.com/paper/2404.06511/) (2024)
- [LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agents](https://mlsanity.com/paper/2503.10200/) (2025)
- [Visual Agentic AI for Spatial Reasoning with a Dynamic API](https://mlsanity.com/paper/2502.06787/) (2025)
- [Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores](https://mlsanity.com/paper/2410.03492/) (2024)
- [LongVideoAgent: Multi-Agent Reasoning with Long Videos](https://mlsanity.com/paper/2512.20618/) (2025)
- [On Randomness in Agentic Evals](https://mlsanity.com/paper/2602.07150/) (2026)
- [Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task](https://mlsanity.com/paper/2512.10359/) (2025)
- [Symphony: A Cognitively-Inspired Multi-Agent System for Long-Video Understanding](https://mlsanity.com/paper/2603.17307/) (2026)
- [SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2512.13874/) (2025)
- [VideoSEAL: Mitigating Evidence Misalignment in Agentic Long Video Understanding by Decoupling Answer Authority](https://mlsanity.com/paper/2605.12571/) (2026)
- [SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaboration](https://mlsanity.com/paper/2604.05079/) (2026)

## Abstract (arXiv)

Agentic Video Question Answering (VideoQA) systems produce answers through adaptive reasoning and tool-use trajectories, yet standard practice evaluates each system once and estimates uncertainty only across questions. This leaves a basic question untested: would the measured method effect survive if the evaluation were run again? We show that it need not. Using Static-SAGE and Dynamic-SAGE as a controlled case study, we repeat the paired comparison twice on identical SAGE-Bench question-video pairs, holding configuration, tool library, and scoring protocol fixed. In the first execution, Dynamic-SAGE outperforms Static-SAGE by +7.33 accuracy points; in the second, the effect reverses to -4.05. Both are individually significant under paired analysis, supporting opposite conclusions. The change in the paired effect between executions is highly significant and far larger than within-execution uncertainty. The reversal is consistent across question format, modality, difficulty, and video duration, and both evaluation arms move significantly. Motivated by this failure, we introduce REPAIR (REpeated PAired Inference Reliability), a protocol that repeats the paired comparison and tests whether the method effect changes across executions, separating directional reproducibility from effect-size stability. Applied across accuracy and execution metrics, REPAIR exposes three behaviors-directional reversal, magnitude shift, and effect attenuationand shows that reductions in reasoning turns and visible tool calls do not imply reproducible reductions in primitive computation or latency. The execution-level movement is comparable to, and here larger than, median gain reported by recent agentic VideoQA systems, contextualizing its magnitude without implying those systems are unstable. Significance within a single agentic execution is insufficient evidence that a reported method effect is reproducible.