# ALBA : Reinforcement Learning for Video Object Segmentation

arXiv: https://arxiv.org/abs/2005.13039 · HTML: https://mlsanity.com/paper/2005.13039/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2020.

## Built on by (most cited first, 9 of 9)

- [Opening up Open-World Tracking](https://mlsanity.com/paper/2104.11221/) (2021)
- [Target-Aware Object Discovery and Association for Unsupervised Video Multi-Object Segmentation](https://mlsanity.com/paper/2104.04782/) (2021)
- [Learning to Recommend Frame for Interactive Video Object Segmentation in the Wild](https://mlsanity.com/paper/2103.10391/) (2021)
- [CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition](https://mlsanity.com/paper/2101.07042/) (2021)
- [Implicit Motion-Compensated Network for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2204.02791/) (2022)
- [Co-attention Propagation Network for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2304.03910/) (2023)
- [Hierarchical Graph Pattern Understanding for Zero-Shot VOS](https://mlsanity.com/paper/2312.09525/) (2023)
- [Sketch-based Video Object Segmentation: Benchmark and Analysis](https://mlsanity.com/paper/2311.07261/) (2023)
- [FTIO: Frequent Temporally Integrated Objects](https://mlsanity.com/paper/2508.16183/) (2025)

## Abstract (arXiv)

We consider the challenging problem of zero-shot video object segmentation (VOS). That is, segmenting and tracking multiple moving objects within a video fully automatically, without any manual initialization. We treat this as a grouping problem by exploiting object proposals and making a joint inference about grouping over both space and time. We propose a network architecture for tractably performing proposal selection and joint grouping. Crucially, we then show how to train this network with reinforcement learning so that it learns to perform the optimal non-myopic sequence of grouping decisions to segment the whole video. Unlike standard supervised techniques, this also enables us to directly optimize for the non-differentiable overlap-based metrics used to evaluate VOS. We show that the proposed method, which we call ALBA outperforms the previous stateof-the-art on three benchmarks: DAVIS 2017 [2], FBMS [20] and Youtube-VOS [27].