mlsanityData to 14 Oct 2026

OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs

arXiv 2606.22617 ↗First posted 21 Jun 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have achieved remarkable performance on 2D visual tasks, yet enhancing their spatial intelligence for real-world applications such as Autonomous Vehicles (AV) remains an open challenge. Existing geometry-aware MLLMs typically rely on auxiliary 3D models at inference time, introducing pipeline complexity and the risk of cascading failures. In this paper, we present OmniSpace, a simple yet effective plug-and-play paradigm for geometry-aware spatial reasoning from purely 2D observations. Motivated by our finding that current MLLMs are bottlenecked by weak cross-view correspondence and depth estimation, OmniSpace introduces a Camera Pose Injector, a Multi-view Epipolar Attention module, and a 3D Geometric Distillation objective that jointly address these two limitations by transferring geometric knowledge into the model. Extensive experiments show that OmniSpace surpasses existing methods on planning benchmarks (nuScenes, Bench2Drive), risk detection (nuInstruct), language (Omnidrive), and generalization (DriveBench).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 71 of its 81 references that are in the map. First posted on arXiv 21 Jun 2026.

Builds on

71 of 81 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views21 Oct
  2. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes8 Sep
  3. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding2 Jun
  4. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors30 May
  5. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence29 May
  6. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec

Other references · 65

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2025Qwen3-VL Technical Reportcited by 2,793
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  11. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  12. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
38 more
  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  2. 2022Planning-oriented Autonomous Drivingcited by 1,087
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  4. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  5. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  6. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  7. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  8. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  9. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  10. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  11. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  12. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  13. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 332
  14. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 252
  15. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  16. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 241
  17. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  18. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  19. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  20. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  21. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 178
  22. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 169
  23. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  24. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  25. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
  26. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 150
  27. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 144
  28. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 143
  29. 2025Cambrian-S: Towards Spatial Supersensing in Videocited by 134
  30. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 126
  31. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
  32. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 86
  33. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 86
  34. 2025Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Drivingcited by 84
  35. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 76
  36. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  37. 2024MAN TruckScenes: A multimodal dataset for autonomous trucking in diverse conditionscited by 41
  38. 2025HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decodercited by 35

50 most cited of 65 All 65 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Camera
  2. 2025OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision
  3. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
  4. 2026Perception-Aware Multimodal Spatial Reasoning from Monocular Images
  5. 2025Are VLMs Ready for Lane Topology Awareness in Autonomous Driving?
  6. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision
  7. 2026Predicting When to Trust Vision-Language Models for Spatial Reasoning
  8. 2025InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models