mlsanityData to 14 Oct 2026

VTO: Visual Tool Orchestration for Video Anomaly Detection

arXiv 2608.08219 ↗First posted 8 Aug 2026cs.CV cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Video anomaly detection (VAD) is a critical yet challenging task due to the complex and diverse nature of real-world scenarios. Traditional deep learning approaches are fundamentally limited by poor generalization across diverse scenarios. While multimodal agents offer a promising tool-learning paradigm for VAD, current systems relying on supervised fine-tuning struggle with complex orchestration, and standard reinforcement learning often causes premature termination due to coarse-grained outcome rewards. To address these challenges, we propose VTO, a process-supervised reinforcement learning framework. Moving beyond static tool usage, VTO enables the agent to dynamically explore and interact with the environment. Specifically, we introduce a foundation model-driven cognitive evaluator to provide context-aware semantic feedback, which is seamlessly integrated into a Process-Supervised Cognitive Alignment that delivers fine-grained, step-wise supervision. By explicitly penalizing logical truncation and rewarding complete causal chains, the agent optimizes its multi-step reasoning policy for interrelated tool orchestration. To support our proposed framework, we meticulously crafted VAD-Tool, a hierarchical visual tool set comprising 12 specialized vision tools spanning from entity tracking to high-stakes hazard detection, and established the corresponding benchmark for rigorous multi-step reasoning evaluation. Extensive experiments on VAD-Tool demonstrate that VTO significantly outperforms baselines, achieving up to a 10.2% absolute accuracy improvement in tool scheduling. Code and data are available at https://github.com/MICLAB-BUPT/VTO.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 56 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

27 of 56 references.

References · 27

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
  5. 2018Real-world Anomaly Detection in Surveillance Videoscited by 577
  6. 2019Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detectioncited by 564
  7. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  8. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  9. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  10. 2016Learning Temporal Regularity in Video Sequencescited by 254
  11. 2020FastReID: A Pytorch Toolbox for General Instance Re-identificationcited by 166
  12. 2024EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithmscited by 113
15 more
  1. 2022An End-to-End Transformer Model for Crowd Localizationcited by 74
  2. 2024Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularitycited by 70
  3. 2024Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Modelscited by 57
  4. 2024Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usagecited by 51
  5. 2024Advancing Video Anomaly Detection: A Concise Review and a New Datasetcited by 49
  6. 2024T2SG: Traffic Topology Scene Graph for Topology Reasoning in Autonomous Drivingcited by 35
  7. 2024Tool-Planner: Task Planning with Clusters across Multiple Toolscited by 33
  8. 2025Global-Local Tree Search in VLMs for 3D Indoor Scene Generationcited by 29
  9. 2025Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regressioncited by 21
  10. 2025Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoningcited by 20
  11. 2025PANDA: Towards Generalist Video Anomaly Detection via Agentic AI Engineercited by 19
  12. 2025Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratchcited by 17
  13. 2025DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistencycited by 15
  14. 2025Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoningcited by 14
  15. 2026VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chainingcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning
  2. 2025Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought
  3. 2026AgenticVAU: Multi-Agent Explore-Verify Reasoning for Video Anomaly Understanding
  4. 2026Anomaly-Guided Observation and Two-Level Semantic Feedback for Training-Free Video Anomaly Analysis
  5. 2026Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding
  6. 2026COPRA: Conditional Parameter Adaptation with Reinforcement Learning for Video Anomaly Detection
  7. 2026Glance, Scrutinize, and Think: Advancing Video Anomaly Detection from Training-Free to Agentic Reasoning
  8. 2026O-VAD: Industrial Video Anomaly Detection through Object-Centric Tracking and Reasoning