mlsanityData to 14 Oct 2026

Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis

arXiv 2312.08782 ↗First posted 14 Dec 2023cs.RO cs.AI cs.CVcited by 103

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building general-purpose robots that operate seamlessly in any environment, with any object, and utilizing various skills to complete diverse tasks has been a long-standing goal in Artificial Intelligence. However, as a community, we have been constraining most robotic systems by designing them for specific tasks, training them on specific datasets, and deploying them within specific environments. These systems require extensively-labeled data and task-specific models. When deployed in real-world scenarios, such systems face several generalization issues and struggle to remain robust to distribution shifts. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models (i.e., foundation models) in research fields such as Natural Language Processing (NLP) and Computer Vision (CV), we devote this survey to exploring (i) how these existing foundation models from NLP and CV can be applied to the field of general-purpose robotics, and also exploring (ii) what a robotics-specific foundation model would look like. We begin by providing a generalized formulation of how foundation models are used in robotics, and the fundamental barriers to making generalist robots universally applicable. Next, we establish a taxonomy to discuss current work exploring ways to leverage existing foundation models for robotics and develop ones catered to robotics. Finally, we discuss key challenges and promising future directions in using foundation models for enabling general-purpose robotic systems. We encourage readers to view our living GitHub repository 2 of resources, including papers reviewed in this survey, as well as related projects and repositories for developing foundation models for robotics.

Key facts. Cited by 103 papers in the map (data through 14 Oct 2026). Builds on 281 of its 374 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

281 of 374 references.

References · 281

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2023Segment Anythingcited by 8,234
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
38 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2022Generative Adversarial Networkscited by 5,627
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  13. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  14. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  15. 2018YOLOv3: An Incremental Improvementcited by 2,659
  16. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  17. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  18. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  19. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  20. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  21. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  22. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  23. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  24. 2016Generative Adversarial Imitation Learningcited by 1,597
  25. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  26. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  27. 2017Adversarial Discriminative Domain Adaptationcited by 1,311
  28. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  29. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  30. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  31. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  32. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  33. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  34. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  35. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 988
  36. 2018DeepMind Control Suitecited by 956
  37. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  38. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886

50 most cited of 281 All 281 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 36; 2025: 45; 2026: 22.

103 papers.

Sort:

By area:

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  2. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  3. 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 314
  4. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  5. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  6. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  7. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  8. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  9. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  10. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  11. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 78
  12. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
38 more
  1. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  2. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 56
  3. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  4. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 52
  5. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  6. 2024STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learningcited by 44
  7. 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 38
  8. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  9. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 36
  10. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  11. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  12. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  13. 2025SRT-H: A Hierarchical Framework for Autonomous Surgery via Language Conditioned Imitation Learningcited by 26
  14. 2024Materials science in the era of large language models: a perspectivecited by 24
  15. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  16. 2025RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Explorationcited by 23
  17. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  18. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 22
  19. 2025Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skillscited by 22
  20. 2025OmniVGGT: Omni-Modality Driven Visual Geometry Grounded Transformercited by 22
  21. 2024VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulationcited by 21
  22. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  23. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 20
  24. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  25. 2024Verifiably Following Complex Robot Instructions with Foundation Modelscited by 16
  26. 2024General-purpose foundation models for increased autonomy in robot-assisted surgerycited by 15
  27. 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systemscited by 15
  28. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  29. 2024ACT-Bench: Towards Action Controllable World Models for Autonomous Drivingcited by 14
  30. 2025AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvementcited by 14
  31. 2024SafeEmbodAI: a Safety Framework for Mobile Robots in Embodied AI Systemscited by 13
  32. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
  33. 2024From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Modelscited by 11
  34. 2025VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibilitycited by 11
  35. 2024Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challengecited by 9
  36. 2025Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learningcited by 8
  37. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguitycited by 8
  38. 2024Deep Reinforcement Learning for Bipedal Locomotion: A Brief Surveycited by 7

50 most cited of 103 All 103 (JSON)