# Collaborative Multi-Agent Video Fast-Forwarding

arXiv: https://arxiv.org/abs/2305.17569 · HTML: https://mlsanity.com/paper/2305.17569/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 71 references that are in the map.
- First posted on arXiv 27 May 2023.

## Builds on (most cited first, 26 of 26)

- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization](https://mlsanity.com/paper/1404.6264/) (2014)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [AdaFrame: Adaptive Frame Selection for Fast Video Recognition](https://mlsanity.com/paper/1811.12432/) (2018)
- [End-to-end Learning of Action Detection from Frame Glimpses in Videos](https://mlsanity.com/paper/1511.06984/) (2015)
- [Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition](https://mlsanity.com/paper/1907.13369/) (2019)
- [Deep Reinforcement Learning-based Image Captioning with Embedding Reward](https://mlsanity.com/paper/1704.03899/) (2017)
- [Video Summarization by Learning from Unpaired Data](https://mlsanity.com/paper/1805.12174/) (2018)
- [Collaborative Summarization of Topic-Related Videos](https://mlsanity.com/paper/1706.03114/) (2017)
- [Human Pose Estimation using Deep Consensus Voting](https://mlsanity.com/paper/1603.08212/) (2016)
- [FFNet: Video Fast-Forwarding via Reinforcement Learning](https://mlsanity.com/paper/1805.02792/) (2018)
- [PoseAgent: Budget-Constrained 6D Object Pose Estimation via Reinforcement Learning](https://mlsanity.com/paper/1612.03779/) (2016)
- [Collaborative Deep Reinforcement Learning for Joint Object Search](https://mlsanity.com/paper/1702.05573/) (2017)
- [Multi-Stream Dynamic Video Summarization](https://mlsanity.com/paper/1812.00108/) (2018)
- [EgoSampling: Fast-Forward and Stereo for Egocentric Videos](https://mlsanity.com/paper/1412.3596/) (2014)
- [Multi-View Surveillance Video Summarization via Joint Embedding and Sparse Optimization](https://mlsanity.com/paper/1706.03121/) (2017)
- [Straight to the Point: Fast-forwarding Videos via Reinforcement Learning Using Textual Data](https://mlsanity.com/paper/2003.14229/) (2020)
- [Fast-Forward Video Based on Semantic Extraction](https://mlsanity.com/paper/1708.04160/) (2017)
- [Learning Visual Storylines with Skipping Recurrent Neural Networks](https://mlsanity.com/paper/1604.04279/) (2016)
- [EgoSampling: Wide View Hyperlapse from Egocentric Videos](https://mlsanity.com/paper/1604.07741/) (2016)
- [Towards Semantic Fast-Forward and Stabilized Egocentric Videos](https://mlsanity.com/paper/1708.04146/) (2017)
- [Video Summarization in a Multi-View Camera Network](https://mlsanity.com/paper/1608.00310/) (2016)
- [Distributed Multi-agent Video Fast-forwarding](https://mlsanity.com/paper/2008.04437/) (2020)

## Built on by (most cited first, 4 of 4)

- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [LMAgent: A Large-scale Multimodal Agents Society for Multi-user Simulation](https://mlsanity.com/paper/2412.09237/) (2024)
- [Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering](https://mlsanity.com/paper/2311.17331/) (2023)
- [TRIMMER: A New Paradigm for Video Summarization through Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2605.01659/) (2026)

## Abstract (arXiv)

Multi-agent applications have recently gained significant popularity. In many computer vision tasks, a network of agents, such as a team of robots with cameras, could work collaboratively to perceive the environment for efficient and accurate situation awareness. However, these agents often have limited computation, communication, and storage resources. Thus, reducing resource consumption while still providing an accurate perception of the environment becomes an important goal when deploying multi-agent systems. To achieve this goal, we identify and leverage the overlap among different camera views in multi-agent systems for reducing the processing, transmission and storage of redundant/unimportant video frames. Specifically, we have developed two collaborative multi-agent video fast-forwarding frameworks in distributed and centralized settings, respectively. In these frameworks, each individual agent can selectively process or skip video frames at adjustable paces based on multiple strategies via reinforcement learning. Multiple agents then collaboratively sense the environment via either 1) a consensus-based distributed framework called DMVF that periodically updates the fast-forwarding strategies of agents by establishing communication and consensus among connected neighbors, or 2) a centralized framework called MFFNet that utilizes a central controller to decide the fast-forwarding strategies for agents based on collected data. We demonstrate the efficacy and efficiency of our proposed frameworks on a real-world surveillance video dataset VideoWeb and a new simulated driving dataset CarlaSim, through extensive simulations and deployment on an embedded platform with TCP communication. We show that compared with other approaches in the literature, our frameworks achieve better coverage of important frames, while significantly reducing the number of frames processed at each agent.