mlsanityData to 7 Oct 2026

Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision

arXiv 2504.02477 ↗First posted 3 Apr 2025cs.RO cs.CVcited by 37

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Robot vision has greatly benefited from advancements in multimodal fusion techniques and vision-language models (VLMs). We adopt a task-oriented perspective to systematically review the applications and advancements of multimodal fusion methods and VLMs in the field of robot vision. For semantic scene understanding tasks, we categorize fusion approaches into encoder-decoder frameworks, attention-based architectures, and graph neural networks. Meanwhile, we also analyze the architectural characteristics and practical implementations of these fusion strategies in key tasks such as simultaneous localization and mapping (SLAM), 3D object detection, navigation, and manipulation. We compare the evolutionary paths and applicability of VLMs based on large language models (LLMs) with traditional multimodal fusion methods.Additionally, we conduct an in-depth analysis of commonly used datasets, evaluating their applicability and challenges in real-world robotic scenarios. Building on this analysis, we identify key challenges in current research, including cross-modal alignment, efficient fusion, real-time deployment, and domain adaptation. We propose future directions such as self-supervised learning for robust multimodal representations, structured spatial memory and environment modeling to enhance spatial intelligence, and the integration of adversarial robustness and human feedback mechanisms to enable ethically aligned system deployment. Through a comprehensive review, comparative analysis, and forward-looking discussion, we provide a valuable reference for advancing multimodal perception and interaction in robotic vision. A comprehensive list of studies in this survey is available at https://github.com/Xiaofeng-Han-Res/MF-RV.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 240 of its 346 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

240 of 346 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  2. 2024InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models18 Nov
  3. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution4 Nov
  4. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  5. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models17 Jul
  6. 2024Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding13 Jul
  7. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  9. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 231

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2024The Llama 3 Herd of Modelscited by 14,192
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
38 more
  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  13. 2020Longformer: The Long-Document Transformercited by 3,362
  14. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  15. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  16. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  17. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  18. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  19. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  20. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  21. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  22. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  23. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  24. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  25. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  26. 2020Reformer: The Efficient Transformercited by 1,658
  27. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  28. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  29. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  30. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  31. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  32. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  33. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  34. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  35. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  36. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  37. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  38. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075

50 most cited of 231 All 231 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 24; 2026: 13.

37 papers.

Sort:

By area:

  1. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  2. 2026A1: A Fully Transparent Open-Source, Adaptive and Efficient Truncated Vision-Language-Action Modelcited by 8
  3. 2025OmniVLA: Physically-Grounded Multimodal VLA with Unified Multi-Sensor Perception for Robotic Manipulationcited by 7
  4. 2025CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigationcited by 5
  5. 2025A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospectscited by 5
  6. 20253D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answeringcited by 4
  7. 2025Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planningcited by 4
  8. 2026Large Language Model-Assisted UAV Operations and Communications: A Multifaceted Survey and Tutorialcited by 4
  9. 2025LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mambacited by 3
  10. 2025ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Followingcited by 3
  11. 2025Dejavu: Towards Experience Feedback Learning for Embodied Intelligencecited by 3
  12. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
25 more
  1. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniquescited by 2
  2. 2025MV-CoRe: Multimodal Visual-Conceptual Reasoning for Complex Visual Question Answeringcited by 2
  3. 2025QuadKAN: KAN-Enhanced Quadruped Motion Control via End-to-End Reinforcement Learningcited by 2
  4. 2025SAGE: Spatial-visual Adaptive Graph Exploration for Efficient Visual Place Recognitioncited by 2
  5. 2026SEF-MAP: Subspace-Decomposed Expert Fusion for Robust Multimodal HD Map Predictioncited by 2
  6. 2025Beyond Single Models: Enhancing LLM Detection of Ambiguity in Requests through Debatecited by 1
  7. 2025CurriFlow: Curriculum-Guided Depth Fusion with Optical Flow-Based Temporal Alignment for 3D Semantic Scene Completioncited by 1
  8. 2025Complementary Information Guided Occupancy Prediction via Multi-Level Representation Fusioncited by 1
  9. 2025Physics-Based Benchmarking Metrics for Multimodal Synthetic Imagescited by 1
  10. 2026CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learningcited by 1
  11. 2026MuseVLA: An Adaptive Multimodal Sensing Vision-Language-Action Model for Robotic Manipulationcited by 1
  12. 2026Invertible Neural Network Adapter for One-Step Flow Matching in Robot Manipulationcited by 1
  13. 2025MM-FusionNet: Context-Aware Dynamic Fusion for Multi-modal Fake News Detection with Large Vision-Language Modelscited by 0
  14. 2025Vision-Proprioception Fusion with Mamba2 in End-to-End Reinforcement Learning for Motion Controlcited by 0
  15. 2025Orchestrate, Generate, Reflect: A VLM-Based Multi-Agent Collaboration Framework for Automated Driving Policy Learningcited by 0
  16. 2025OceanGym: A Benchmark Environment for Underwater Embodied Agentscited by 0
  17. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomycited by 0
  18. 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agentscited by 0
  19. 2026MMP-A*: Multimodal Perception Enhanced Incremental Heuristic Search on Path Planningcited by 0
  20. 2026Evaluation as Evolution: Transforming Adversarial Diffusion into Closed-Loop Curricula for Autonomous Vehiclescited by 0
  21. 2026Efficient Semantic Image Communication for Traffic Monitoring at the Edgecited by 0
  22. 2026Twincher: Bijective Representation Learning for Robust Inversion of Continuous Systemscited by 0
  23. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generationcited by 0
  24. 2026AdaptVPR: Route-Aware Hard Positive Generation for Robust Visual Place Recognitioncited by 0
  25. 2026Safe Multi-Robot Coordination via VLM-LLM Reasoning and Reachability Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HMVLA: Hyperbolic Multimodal Fusion for Vision-Language-Action Models
  2. 2025Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs
  3. 2023RoboLLM: Robotic Vision Tasks Grounded on Multimodal Large Language Models
  4. 2026Selective Perception for Robot: Task-Aware Attention in Multimodal VLA
  5. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
  6. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
  7. 2025Small Vision-Language Models: A Survey on Compact Architectures and Techniques
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks