mlsanityData to 7 Oct 2026

Real-World Robot Applications of Foundation Models: A Review

arXiv 2402.05741 ↗First posted 8 Feb 2024cs.RO cs.AI cs.CVcited by 60

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent developments in foundation models, like Large Language Models (LLMs) and Vision-Language Models (VLMs), trained on extensive data, facilitate flexible application across different tasks and modalities. Their impact spans various fields, including healthcare, education, and robotics. This paper provides an overview of the practical application of foundation models in real-world robotics, with a primary emphasis on the replacement of specific components within existing robot systems. The summary encompasses the perspective of input-output relationships in foundation models, as well as their role in perception, motion planning, and control within the field of robotics. This paper concludes with a discussion of future challenges and implications for practical robot applications.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 193 of its 241 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

193 of 241 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  2. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectives9 Jan
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  4. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition20 Mar
  5. 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors10 Mar
  6. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  7. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar
  8. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models21 Nov
  9. 2022Visual Language Maps for Robot Navigation11 Oct

Other references · 184

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2023GPT-4 Technical Reportcited by 18,132
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  12. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
38 more
  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2023Segment Anythingcited by 8,158
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  13. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  14. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  15. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  16. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  17. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  18. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  19. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  20. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  21. 2015VQA: Visual Question Answeringcited by 2,489
  22. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  23. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  24. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  25. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  26. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  27. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  28. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  29. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  30. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  31. 2023Mastering Diverse Domains through World Modelscited by 1,193
  32. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  33. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  34. 2023PaLM 2 Technical Reportcited by 1,154
  35. 2021Grounded Language-Image Pre-trainingcited by 1,147
  36. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  37. 2022Human Motion Diffusion Modelcited by 1,062
  38. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057

50 most cited of 184 All 184 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 32; 2026: 10.

60 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  3. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  4. 2025Leave No Observation Behind: Real-time Correction for VLA Action Chunkscited by 45
  5. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  6. 2024EVA: An Embodied World Model for Future Video Anticipationcited by 28
  7. 2025Large Language Models for Multi-Robot Systems: A Surveycited by 26
  8. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  9. 2024VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulationcited by 21
  10. 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interactioncited by 18
  11. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  12. 2024VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robotscited by 16
38 more
  1. 2024Generative Emergent Communication: Large Language Model is a Collective World Modelcited by 15
  2. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  3. 2024Towards Artwork Explanation in Large-scale Vision Language Modelscited by 8
  4. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguitycited by 8
  5. 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Modelscited by 8
  6. 2026FutureRTC: Real-Time Robot Execution with Anticipatory-Conditioned Action Chunkingcited by 8
  7. 2025KeyWorld: Key Frame Reasoning Enables Effective and Efficient World Modelscited by 7
  8. 2026From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulationcited by 6
  9. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
  10. 2025Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulationcited by 5
  11. 2025IMPACT: Intelligent Motion Planning with Acceptable Contact Trajectories via Vision-Language Modelscited by 4
  12. 2024Foundation Models for Autonomous Robots in Unstructured Environmentscited by 3
  13. 2024DM2RM: Dual-Mode Multimodal Ranking for Target Objects and Receptacles Based on Open-Vocabulary Instructionscited by 3
  14. 2024Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representationscited by 3
  15. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
  16. 2024Towards an LLM-Based Speech Interface for Robot-Assisted Feedingcited by 2
  17. 2024Advances in Transformers for Robotic Applications: A Reviewcited by 2
  18. 2024Do large language vision models understand 3D shapes?cited by 2
  19. 2025System 0/1/2/3: Quad-process theory for multi-timescale embodied collective cognitive systemscited by 2
  20. 2025Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusioncited by 2
  21. 2025Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Researchcited by 2
  22. 2026E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learningcited by 2
  23. 2026Hijacking Robots with a Piece of Paper: A Systematic Study of Physical Prompt Injection in VLM-Controlled Robotscited by 2
  24. 2024Object Segmentation from Open-Vocabulary Manipulation Instructions Based on Optimal Transport Polygon Matching with Multimodal Foundation Modelscited by 1
  25. 2025Tonguescape: Exploring Language Models Understanding of Vowel Articulationcited by 1
  26. 2025APEX: Empowering LLMs with Physics-Based Task Planning for Real-time Insightcited by 1
  27. 2025INTENTION: Inferring Tendencies of Humanoid Robot Motion Through Interactive Intuition and Grounded VLMcited by 1
  28. 2025Human Centric General Physical Intelligence for Agile Manufacturing Automationcited by 1
  29. 2025Robot-Powered Data Flywheels: Deploying Robots in the Wild for Continual Data Collection and Foundation Model Adaptationcited by 1
  30. 2025One Patch is All You Need: Joint Surface Material Reconstruction and Classification from Minimal Visual Cuescited by 1
  31. 2025Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interactioncited by 1
  32. 2025SAGA: Open-World Mobile Manipulation via Structured Affordance Groundingcited by 1
  33. 2026Flow as Flow: Modeling Robot Velocity Fields as Probability Velocity Fields for Flow-Based Object Manipulationcited by 1
  34. 2024Mitigating Downstream Model Risks via Model Provenancecited by 0
  35. 2024RAMPA: Robotic Augmented Reality for Machine Programming by DemonstrAtioncited by 0
  36. 2024Robotic State Recognition with Image-to-Text Retrieval Task of Pre-Trained Vision-Language Model and Black-Box Optimizationcited by 0
  37. 2025MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systemscited by 0
  38. 2025Search-Based Software Engineering and AI Foundation Models: Current Landscape and Future Roadmapcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Foundation Model Driven Robotics: A Comprehensive Review
  2. 2026Foundation Models in Robotics: A Comprehensive Review of Methods, Models, Datasets, Challenges and Future Research Directions
  3. 2025From Grounding to Manipulation: Case Studies of Foundation Model Integration in Embodied Robotic Systems
  4. 2026The Embodiment Gap in Robot Foundation Models
  5. 2024What Matters in Building Vision-Language-Action Models for Generalist Robots
  6. 2026Are Foundation Models the Route to Full-Stack Transfer in Robotics? ↗
  7. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks
  8. 2026Vision-Language-Action Models: Experimental Insights from a Real-World UR5 Platform