# Multi-Agent Self-Improving Reinforcement Learning for Video Reasoning

arXiv: https://arxiv.org/abs/2608.28675 · HTML: https://mlsanity.com/paper/2608.28675/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 46 of its 58 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Builds on (most cited first, 30 of 46)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [VTimeLLM: Empower LLM to Grasp Video Moments](https://mlsanity.com/paper/2311.18445/) (2023)
- [VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2504.06958/) (2025)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [Valley: Video Assistant with Large Language model Enhanced abilitY](https://mlsanity.com/paper/2306.07207/) (2023)
- [V-STaR: Training Verifiers for Self-Taught Reasoners](https://mlsanity.com/paper/2402.06457/) (2024)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Can I Trust Your Answer? Visually Grounded Video Question Answering](https://mlsanity.com/paper/2309.01327/) (2023)
- [Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning](https://mlsanity.com/paper/2402.11435/) (2024)

## Abstract (arXiv)

Video reasoning tasks such as grounded video question answering and temporal grounding require selecting temporal evidence that supports the query. In many current training setups, temporal supervision is applied through local objectives such as boundary regression or span generation, while verification is used mainly to rerank candidate segments at inference time. We study whether a frozen verifier can also guide training. Our multi-agent framework couples a trainable Grounder with a frozen Verifier: the Grounder samples candidate trajectories and evidence segments, the Verifier assigns query-conditioned segment scores, a group-relative policy-gradient objective favors trajectories that outperform their within-input peers, and a bootstrapped calibration loss steers temporal predictions toward verifier-preferred spans. Trained on source tasks and evaluated without target-dataset fine-tuning, a two-billion-parameter instantiation transfers zero-shot across grounded question answering, temporal grounding, and long-video question answering, reaching 28.7% intersection-over-union and 25.4% answer-grounding accuracy on a grounded-question-answering benchmark, 46.1% intersection-over-union on a temporal-grounding benchmark, and 54.1% on a long-video question-answering benchmark. Relative to a strong same-scale baseline, the gains are modest but consistent, with the clearest improvements on relevance-oriented metrics such as intersection-over-union and moderate-overlap recall. Within the tested benchmarks and transfer setting, the results support frozen verification as a training signal for evidence selection, while showing that strict boundary precision remains comparatively weaker. Code and models are available at https://anonymous.4open.science/r/MASIRL-E50C/