# Leader360V: The Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment arXiv: https://arxiv.org/abs/2506.14271 ยท HTML: https://mlsanity.com/paper/2506.14271/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 43 of its 56 references that are in the map. - First posted on arXiv 17 Jun 2025. ## Builds on (most cited first, 30 of 43) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016) - [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022) - [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024) - [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021) - [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021) - [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016) - [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022) - [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018) - [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018) - [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018) - [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022) - [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022) - [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023) - [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023) - [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021) - [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021) - [Panacea: Panoramic and Controllable Video Generation for Autonomous Driving](https://mlsanity.com/paper/2311.16813/) (2023) - [TAO: A Large-Scale Benchmark for Tracking Any Object](https://mlsanity.com/paper/2005.10356/) (2020) - [AiATrack: Attention in Attention for Transformer Visual Tracking](https://mlsanity.com/paper/2207.09603/) (2022) - [Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking](https://mlsanity.com/paper/2203.05328/) (2022) - [HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions](https://mlsanity.com/paper/2207.14284/) (2022) - [High-Quality Entity Segmentation](https://mlsanity.com/paper/2211.05776/) (2022) - [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023) - [LVOS: A Benchmark for Long-term Video Object Segmentation](https://mlsanity.com/paper/2211.10181/) (2022) - [OMG-Seg: Is One Model Good Enough For All Segmentation?](https://mlsanity.com/paper/2401.10229/) (2024) - [Waymo Open Dataset: Panoramic Video Panoptic Segmentation](https://mlsanity.com/paper/2206.07704/) (2022) - [360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model](https://mlsanity.com/paper/2401.06578/) (2024) ## Built on by (most cited first, 8 of 8) - [OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedback](https://mlsanity.com/paper/2511.00510/) (2025) - [Sekai2: From World Exploration to Interactive World Modeling](https://mlsanity.com/paper/2608.09449/) (2026) - [AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video](https://mlsanity.com/paper/2609.14462/) (2026) - [PanoSAM2: Lightweight Distortion- and Memory-aware Adaptions of SAM2 for 360 Video Object Segmentation](https://mlsanity.com/paper/2604.07901/) (2026) - [Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modeling](https://mlsanity.com/paper/2606.27745/) (2026) - [AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2608.29081/) (2026) - [Stitch4D: Sparse Multi-Location 4D Urban Reconstruction via Spatio-Temporal Interpolation](https://mlsanity.com/paper/2604.07923/) (2026) - [MUGEN: Interactive Panoramic World Exploration via Camera Control](https://mlsanity.com/paper/2609.38077/) (2026) ## Abstract (arXiv) 360 video captures the complete surrounding scenes with the ultra-large field of view of 360X180. This makes 360 scene understanding tasks, eg, segmentation and tracking, crucial for appications, such as autonomous driving, robotics. With the recent emergence of foundation models, the community is, however, impeded by the lack of large-scale, labelled real-world datasets. This is caused by the inherent spherical properties, eg, severe distortion in polar regions, and content discontinuities, rendering the annotation costly yet complex. This paper introduces Leader360V, the first large-scale, labeled real-world 360 video datasets for instance segmentation and tracking. Our datasets enjoy high scene diversity, ranging from indoor and urban settings to natural and dynamic outdoor scenes. To automate annotation, we design an automatic labeling pipeline, which subtly coordinates pre-trained 2D segmentors and large language models to facilitate the labeling. The pipeline operates in three novel stages. Specifically, in the Initial Annotation Phase, we introduce a Semantic- and Distortion-aware Refinement module, which combines object mask proposals from multiple 2D segmentors with LLM-verified semantic labels. These are then converted into mask prompts to guide SAM2 in generating distortion-aware masks for subsequent frames. In the Auto-Refine Annotation Phase, missing or incomplete regions are corrected either by applying the SDR again or resolving the discontinuities near the horizontal borders. The Manual Revision Phase finally incorporates LLMs and human annotators to further refine and validate the annotations. Extensive user studies and evaluations demonstrate the effectiveness of our labeling pipeline. Meanwhile, experiments confirm that Leader360V significantly enhances model performance for 360 video segmentation and tracking, paving the way for more scalable 360 scene understanding.