mlsanityData to 14 Oct 2026

Dense360: Dense Understanding from Omnidirectional Panoramas

arXiv 2506.14471 ↗First posted 17 Jun 2025cs.CVcited by 9

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) require comprehensive visual inputs to achieve dense understanding of the physical world. While existing MLLMs demonstrate impressive world understanding capabilities through limited field-of-view (FOV) visual inputs (e.g., 70 degree), we take the first step toward dense understanding from omnidirectional panoramas. We first introduce an omnidirectional panoramas dataset featuring a comprehensive suite of reliability-scored annotations. Specifically, our dataset contains 160K panoramas with 5M dense entity-level captions, 1M unique referring expressions, and 100K entity-grounded panoramic scene descriptions. Compared to multi-view alternatives, panoramas can provide more complete, compact, and continuous scene representations through equirectangular projections (ERP). However, the use of ERP introduces two key challenges for MLLMs: i) spatial continuity along the circle of latitude, and ii) latitude-dependent variation in information density. We address these challenges through ERP-RoPE, a position encoding scheme specifically designed for panoramic ERP. In addition, we introduce Dense360-Bench, the first benchmark for evaluating MLLMs on omnidirectional captioning and grounding, establishing a comprehensive framework for advancing dense visual-language understanding in panoramic settings.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 70 of its 77 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

70 of 77 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation24 Apr
  2. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
  3. 2025An Egocentric Vision-Language Model based Portable Real-time Smart Assistant6 Mar
  4. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigation4 Mar
  5. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigation20 Feb

Other references · 65

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2025Qwen2.5-VL Technical Reportcited by 5,587
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
38 more
  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  9. 2024VMamba: Visual State Space Modelcited by 969
  10. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  11. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  12. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  13. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  14. 2015An Introduction to Convolutional Neural Networkscited by 514
  15. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
  16. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  17. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
  18. 2022Egocentric Video-Language Pretrainingcited by 232
  19. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  20. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  21. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  22. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 113
  23. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  24. 2022High-Quality Entity Segmentationcited by 93
  25. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  26. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  27. 2023DVIS: Decoupled Video Instance Segmentation Frameworkcited by 77
  28. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  29. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  30. 2023Aligning and Prompting Everything All at Once for Universal Visual Perceptioncited by 60
  31. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoningcited by 52
  32. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMscited by 52
  33. 2023DVIS++: Improved Decoupled Framework for Universal Video Segmentationcited by 46
  34. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 41
  35. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  36. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 37
  37. 2024360+x: A Panoptic Multi-modal Scene Understanding Datasetcited by 35
  38. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025ODI-Bench: Can MLLMs Understand Immersive Omnidirectional Environments?cited by 12
  2. 2026PanoWorld: Towards Spatial Supersensing in 360^circ Panorama Worldcited by 5
  3. 2026Panoramic Affordance Predictioncited by 1
  4. 2026Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modelingcited by 1
  5. 2026EAGLE-360: Embodied Active Global-to-Local Exploration in 360^circcited by 1
  6. 2025JoPano: Unified Panorama Generation via Joint Modelingcited by 0
  7. 2026OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoningcited by 0
  8. 2026Seek to Segment: Active Perception for Panoramic Referring Segmentationcited by 0
  9. 2026SCCM: Spherically Consistent Coarse Matching for ERP Dense Feature Correspondencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026More than the Sum: Panorama-Language Models for Adverse Omni-Scenes
  2. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World
  3. 2025OpenView: Empowering MLLMs with Out-of-view VQA
  4. 2025Towards Omnidirectional Reasoning with 360-R1: A Dataset, Benchmark, and GRPO-based Method
  5. 2025DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotation
  6. 2025Omnidirectional Spatial Modeling from Correlated Panoramas
  7. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception
  8. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding