mlsanityData to 7 Oct 2026

A Survey on Robotics with Foundation Models: toward Embodied AI

arXiv 2402.02385 ↗First posted 4 Feb 2024cs.RO cs.AIcited by 53

Not on the map. Why?

Abstract (arXiv)

While the exploration for embodied AI has spanned multiple decades, it remains a persistent challenge to endow agents with human-level intelligence, including perception, learning, reasoning, decision-making, control, and generalization capabilities, so that they can perform general-purpose tasks in open, unstructured, and dynamic environments. Recent advances in computer vision, natural language processing, and multi-modality learning have shown that the foundation models have superhuman capabilities for specific tasks. They not only provide a solid cornerstone for integrating basic modules into embodied AI systems but also shed light on how to scale up robot learning from a methodological perspective. This survey aims to provide a comprehensive and up-to-date overview of foundation models in robotics, focusing on autonomous manipulation and encompassing high-level planning and low-level control. Moreover, we showcase their commonly used datasets, simulators, and benchmarks. Importantly, we emphasize the critical challenges intrinsic to this field and delineate potential avenues for future research, contributing to advancing the frontier of academic and industrial discourse.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 57 of its 63 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

57 of 63 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
38 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  8. 2022Red Teaming Language Models with Language Modelscited by 983
  9. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  10. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  11. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  12. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  13. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  14. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  15. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  16. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  17. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  18. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  19. 2023A Cookbook of Self-Supervised Learningcited by 248
  20. 2023Track Anything: Segment Anything Meets Videoscited by 230
  21. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  22. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  23. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  24. 2023Structured World Models from Human Videoscited by 144
  25. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  26. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  27. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  28. 2022A Comprehensive Survey of Image Augmentation Techniques for Deep Learningcited by 95
  29. 2023CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalizationcited by 88
  30. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 79
  31. 2023ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planningcited by 77
  32. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  33. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  34. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  35. 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
  36. 2023What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discoverycited by 22
  37. 2023Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Trainingcited by 18
  38. 2024Language-Conditioned Robotic Manipulation with Fast and Slow Thinkingcited by 17

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 32; 2026: 10.

53 papers.

Sort:

By area:

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  3. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  4. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  5. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  6. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  7. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
  8. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 40
  9. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  10. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  11. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  12. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
38 more
  1. 2025Embodied AI: From LLMs to World Modelscited by 23
  2. 2025DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Modelscited by 17
  3. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  4. 2025SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agentscited by 15
  5. 2024Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robotscited by 14
  6. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agentscited by 13
  7. 2025Reinforced Reasoning for Embodied Planningcited by 12
  8. 2025PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Eracited by 12
  9. 2024Privacy-Preserving Video Anomaly Detection: A Surveycited by 11
  10. 2025Online Reasoning Video Segmentation with Just-in-Time Digital Twinscited by 11
  11. 2024Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Gamescited by 8
  12. 2024LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulationcited by 8
  13. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  14. 2026Inject Once Survive Later: Backdooring Vision-Language-Action Models to Persist Through Downstream Fine-tuningcited by 7
  15. 2025CALM: Curiosity-Driven Auditing for Large Language Modelscited by 6
  16. 2025Refer to Any Segmentation Mask Group With Vision-Language Promptscited by 5
  17. 2025DropVLA: An Action-Level Backdoor Attack on Vision-Language-Action Modelscited by 5
  18. 2026Open-Loop Planning, Closed-Loop Verification: Speculative Verification for VLAcited by 5
  19. 2026ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AIcited by 5
  20. 2024Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Followingcited by 3
  21. 2025Unraveling Human-AI Teaming: A Review and Outlookcited by 3
  22. 2025TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-makingcited by 3
  23. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  24. 2025Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulationcited by 3
  25. 2025Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusioncited by 2
  26. 2025Fusing Cross-Domain Knowledge from Multimodal Data to Solve Problems in the Physical Worldcited by 2
  27. 2025Multimodal Data Storage and Retrieval for Embodied AI: A Surveycited by 2
  28. 2025ANNIE: Be Careful of Your Robotscited by 2
  29. 2025Adversarially-Refined VQ-GAN with Dense Motion Tokenization for Spatio-Temporal Heatmapscited by 2
  30. 2026Self-evolving Embodied AIcited by 2
  31. 2024PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLMcited by 1
  32. 2025Offline Diffusion Policy for Multi-User Delay-Constrained Schedulingcited by 1
  33. 2025FlashDP: Private Training Large Language Models with Efficient DP-SGDcited by 1
  34. 2025RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learningcited by 1
  35. 2026SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agentscited by 1
  36. 2025VIVA+: Human-Centered Situational Decision-Makingcited by 0
  37. 2026MVR: Multi-view Video Reward Shaping for Reinforcement Learningcited by 0
  38. 2026VLAMotor: Test-Guided Enhancement of Vision-Language-Action Models via Agent-BasedData Synthesiscited by 0

50 most cited of 53 All 53 (JSON)