# PanoSAM2: Lightweight Distortion- and Memory-aware Adaptions of SAM2 for 360 Video Object Segmentation

arXiv: https://arxiv.org/abs/2604.07901 · HTML: https://mlsanity.com/paper/2604.07901/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 47 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Builds on (most cited first, 30 of 39)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://mlsanity.com/paper/2306.00989/) (2023)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [Panacea: Panoramic and Controllable Video Generation for Autonomous Driving](https://mlsanity.com/paper/2311.16813/) (2023)
- [Learning Video Object Segmentation from Static Images](https://mlsanity.com/paper/1612.02646/) (2016)
- [Collaborative Video Object Segmentation by Foreground-Background Integration](https://mlsanity.com/paper/2003.08333/) (2020)
- [Video Object Segmentation with Episodic Graph Memory Networks](https://mlsanity.com/paper/2007.07020/) (2020)
- [Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement](https://mlsanity.com/paper/2010.07958/) (2020)
- [Towards Grand Unification of Object Tracking](https://mlsanity.com/paper/2207.07078/) (2022)
- [Collaborative Video Object Segmentation by Multi-Scale Foreground-Background Integration](https://mlsanity.com/paper/2010.06349/) (2020)
- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)
- [SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree](https://mlsanity.com/paper/2410.16268/) (2024)
- [Joint Inductive and Transductive Learning for Video Object Segmentation](https://mlsanity.com/paper/2108.03679/) (2021)
- [XMem++: Production-level Video Segmentation From Few Annotated Frames](https://mlsanity.com/paper/2307.15958/) (2023)
- [Recurrent Dynamic Embedding for Video Object Segmentation](https://mlsanity.com/paper/2205.03761/) (2022)
- [Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning](https://mlsanity.com/paper/2408.07931/) (2024)
- [Robust Visual Tracking by Segmentation](https://mlsanity.com/paper/2203.11191/) (2022)
- [360+x: A Panoptic Multi-modal Scene Understanding Dataset](https://mlsanity.com/paper/2404.00989/) (2024)
- [TarViS: A Unified Approach for Target-based Video Segmentation](https://mlsanity.com/paper/2301.02657/) (2023)
- [Imagine360: Immersive 360 Video Generation from Perspective Anchor](https://mlsanity.com/paper/2412.03552/) (2024)

## Built on by (most cited first, 1 of 1)

- [Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modeling](https://mlsanity.com/paper/2606.27745/) (2026)

## Abstract (arXiv)

360 video object segmentation (360VOS) aims to predict temporally-consistent masks in 360 videos, offering full-scene coverage, benefiting applications, such as VR/AR and embodied AI. Learning 360VOS model is nontrivial due to the lack of high-quality labeled dataset. Recently, Segment Anything Models (SAMs), especially SAM2 -- with its design of memory module -- shows strong, promptable VOS capability. However, directly using SAM2 for 360VOS yields implausible results as 360 videos suffer from the projection distortion, semantic inconsistency of left-right sides, and sparse object mask information in SAM2's memory. To this end, we propose PanoSAM2, a novel 360VOS framework based on our lightweight distortion- and memory-aware adaptation strategies of SAM2 to achieve reliable 360VOS while retaining SAM2's user-friendly prompting design. Concretely, to tackle the projection distortion and semantic inconsistency issues, we propose a Pano-Aware Decoder with seam-consistent receptive fields and iterative distortion refinement to maintain continuity across the 0/360 degree boundary. Meanwhile, a Distortion-Guided Mask Loss is introduced to weight pixels by distortion magnitude, stressing stretched regions and boundaries. To address the object sparsity issue, we propose a Long-Short Memory Module to maintain a compact long-term object pointer to re-instantiate and align short-term memories, thereby enhancing temporal coherence. Extensive experiments show that PanoSAM2 yields substantial gains over SAM2: +5.6 on 360VOTS and +6.7 on PanoVOS, showing the effectiveness of our method.