Autoregressive Universal Video Segmentation Model
Not on the map. Why?
Abstract (arXiv)
Recent video foundation models such as SAM2 excel at prompted video segmentation by treating masks as a general-purpose primitive. However, many real-world settings require unprompted segmentation that aims to detect and track all objects in a video without external cues, leaving today's landscape fragmented across task-specific models and pipelines. We recast streaming video segmentation as sequential mask prediction, analogous to language modeling, and introduce the Autoregressive Universal Segmentation Model (AUSM), a single architecture that unifies both prompted and unprompted video segmentation. Built on recent state-space models, AUSM maintains a fixed-size spatial state and scales to video streams of arbitrary length. Furthermore, all components of AUSM are designed for parallel training across frames, yielding substantial speedups over iterative training. On standard benchmarks (DAVIS17, YouTube-VOS 2018 & 2019, MOSE, YouTube-VIS 2019 & 2021, and OVIS) AUSM outperforms prior universal streaming video segmentation methods and achieves up to 2.5x faster training on 16-frame sequences.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 51 of its 56 references that are in the map. First posted on arXiv 26 Aug 2025.
Builds on
51 of 56 references.
References · 51
- 2017Attention Is All You Needcited by 60,361
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2023Segment Anythingcited by 8,234
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
38 more
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
- 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
- 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
- 2016MOT16: A Benchmark for Multi-Object Trackingcited by 532
- 2021TrackFormer: Multi-Object Tracking with Transformerscited by 518
- 2020End-to-End Video Instance Segmentation with Transformerscited by 443
- 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
- 2019Video Object Segmentation using Space-Time Memory Networkscited by 438
- 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
- 2019Video Instance Segmentationcited by 393
- 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 383
- 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
- 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 234
- 2021Associating Objects with Transformers for Video Object Segmentationcited by 226
- 2019MOTS: Multi-Object Tracking and Segmentationcited by 223
- 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192
- 2021Occluded Video Instance Segmentation: A Benchmarkcited by 154
- 2022Decoupling Features in Hierarchical Propagation for Video Object Segmentationcited by 148
- 2020Video Panoptic Segmentationcited by 142
- 2021Video Instance Segmentation using Inter-Frame Communication Transformerscited by 105
- 2022In Defense of Online Models for Video Instance Segmentationcited by 97
- 2022VITA: Video Instance Segmentation via Object Token Associationcited by 94
- 2022MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Trainingcited by 94
- 2022Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentationcited by 79
- 2023DVIS: Decoupled Video Instance Segmentation Frameworkcited by 77
- 2024Repetition Improves Language Model Embeddingscited by 72
- 2024Just read twice: closing the recall gap for recurrent language modelscited by 60
- 2022A Generalized Framework for Video Instance Segmentationcited by 53
- 2022TubeFormer-DeepLab: Video Mask Transformercited by 35
- 2023TarViS: A Unified Approach for Target-based Video Segmentationcited by 33
- 2024UniVS: Unified and Universal Video Segmentation with Prompts as Queriescited by 27
- 2023UniRef++: Segment Every Reference Object in Spatial and Temporal Spacescited by 22
- 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024ReferEverything: Towards Segmenting Everything We Can Speak of in Videos
- 2025SAM2Auto: Auto Annotation Using FLASH
- 2025Long-Context Autoregressive Video Modeling with Next-Frame Prediction
- 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos
- 2022Efficient Video Segmentation Models with Per-frame Inference
- 2026TinySAM 2: Extreme Memory Compression for Efficient Track Anything Model
- 2025MAGI-1: Autoregressive Video Generation at Scale
- 2025The 1st Solution for 7th LSVOS RVOS Track: SaSaSa2VA