# LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents

arXiv: https://arxiv.org/abs/2608.07585 · HTML: https://mlsanity.com/paper/2608.07585/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Builds on (most cited first, 18 of 18)

- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos](https://mlsanity.com/paper/2405.19209/) (2024)
- [Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs](https://mlsanity.com/paper/2506.22139/) (2025)
- [DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs](https://mlsanity.com/paper/2406.04334/) (2024)
- [BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding](https://mlsanity.com/paper/2503.21483/) (2025)
- [VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning](https://mlsanity.com/paper/2503.13444/) (2025)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understanding](https://mlsanity.com/paper/2508.04369/) (2025)
- [Vision Function Layer in Multimodal LLMs](https://mlsanity.com/paper/2509.24791/) (2025)
- [VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking](https://mlsanity.com/paper/2603.20185/) (2026)
- [Augmenting Multi-Agent Communication with State Delta Trajectory](https://mlsanity.com/paper/2506.19209/) (2025)
- [Video-EM: Event-Centric Episodic Memory for Long-Form Video Understanding](https://mlsanity.com/paper/2508.09486/) (2025)
- [GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding](https://mlsanity.com/paper/2603.25072/) (2026)
- [SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2512.13874/) (2025)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning](https://mlsanity.com/paper/2603.25021/) (2026)
- [Latent Visual Cache for Video Reasoning](https://mlsanity.com/paper/2607.02607/) (2026)

## Abstract (arXiv)

Long-video understanding requires models to efficiently acquire and reuse sparse visual evidence from long and redundant video streams. Recent video tool-use agents address this challenge by iteratively invoking visual Tools at different temporal scales, but their Tool-Planner communication typically relies on textual observations. Such text-only interfaces provide lossy summaries of Tool computations, causing previously computed visual evidence not verbalized to be discarded and unavailable for subsequent planning. We identify this limitation as the Tool observation bottleneck and propose Latent Visual Evidence-Enhanced Planning (LAVE), a training-free framework for reusing latent visual evidence from completed Tool calls. LAVE introduces a dual-channel observation interface: the visible channel preserves the original textual trajectory, while the latent channel stores pre-verbal visual updates with their Tool roles, source-frame timestamps, and visual locations. During planning, LAVE retrieves evidence relevant to the current Planner state but not covered by textual observations, and integrates it through bounded timestamp-aligned latent updates with entropy-constrained frame-time routing. This enables video agents to reuse existing visual computation without additional training, frame replay, or modifications to the original orchestration. Extensive experiments on Video-MME, LongVideoBench, and CG-Bench show that LAVE consistently improves video tool-use agents across backbones. Under a comparable frame budget, LAVE improves the Video-MME overall score by 3.76 points over the strongest baseline, demonstrating the effectiveness of latent visual evidence reuse for multi-step video-agent planning.