# Toward Open-World Video Segmentation over Long Horizons

arXiv: https://arxiv.org/abs/2606.15632 · HTML: https://mlsanity.com/paper/2606.15632/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 42 references that are in the map.
- First posted on arXiv 14 Jun 2026.

## Builds on (most cited first, 6 of 6)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [3AM: 3egment Anything with Geometric Consistency in Videos](https://mlsanity.com/paper/2601.08831/) (2026)
- [Solving the Maximum-Weight Connected Subgraph Problem to Optimality](https://mlsanity.com/paper/1409.5308/) (2014)
- [Redefining Instance Matching: A Unified Framework for Part-Aware Matching in Panoptic Segmentation Evaluation](https://mlsanity.com/paper/2605.31094/) (2026)

## Abstract (arXiv)

Long videos challenge open-world segmentation systems to continually discover objects and preserve their identities through disappearance and reappearance. Savvy, a zero-shot, semi-online, class-agnostic system for persistent object discovery and identity maintenance, and OGA, an evaluation suite that credits coherent part-level predictions when their granularity differs from the reference annotations. Savvy combines modular mask discovery, deferred admission based on accumulated evidence, and track consolidation to maintain an evolving object set. OGA allows multiple predictions to support one reference while preserving prediction IDs, pairing granularity-aware fidelity with diagnostics of identity bleeding and fragmented support. Across 142 ScanNet videos and 106 HM3D trajectories, evaluated using adaptively sampled prefixes capped at 1,500 source frames, Savvy outperforms DEVA+SAM and EntitySAM in VPQ_inf, STQ, and AQ under both conventional and OGA evaluation. Conventional VPQ_inf reaches 18.44 versus DEVA+SAM's 11.52 on ScanNet and 17.31 versus 6.58 on HM3D. VIPSeg comparisons reveal conventional one-to-one VPQ's sensitivity to annotation granularity. Controlled identity-severing and flicker tests show that OGA detects temporal failures even when frame-level masks remain unchanged. Together, these results demonstrate complementary advances: Savvy improves long-horizon segmentation and association, and OGA distinguishes coherent part-level support from temporal identity failures. Project website: https://github.com/QingSuML/savvy