mlsanityData to 14 Oct 2026

RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation

arXiv 2306.11706 ↗First posted 20 Jun 2023cs.RO cs.LGcited by 111

On the map: LLM agents and tool use → Multimodal and embodied agents

also in 1 more
Abstract (arXiv)

The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learning. Inspired by recent advances in foundation models for vision and language, we propose a multi-embodiment, multi-task generalist agent for robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned decision transformer capable of consuming action-labelled visual experience. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With RoboCat, we demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100-1000 examples for the target task. We also show how a trained model itself can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop. We investigate the agent's capabilities, with large-scale evaluations both in simulation and on three different real robot embodiments. We find that as we grow and diversify its training data, RoboCat not only shows signs of cross-task transfer, but also becomes more efficient at adapting to new tasks.

Key facts. Cited by 111 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 53 of its 82 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

53 of 82 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
38 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  3. 2017Decoupled Weight Decay Regularizationcited by 1,831
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  6. 2017Hindsight Experience Replaycited by 1,136
  7. 2018DeepMind Control Suitecited by 956
  8. 2022A Generalist Agentcited by 808
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  12. 2021ImageNet-21K Pretraining for the Massescited by 613
  13. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  14. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  15. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  16. 2018Maximum a Posteriori Policy Optimisationcited by 328
  17. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  18. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  19. 2022Masked Visual Pre-training for Motor Controlcited by 248
  20. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  21. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  22. 2022Multi-Game Decision Transformerscited by 214
  23. 2021Alignment of Language Agentscited by 168
  24. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  25. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  26. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  27. 2019Optimal Policies Tend to Seek Powercited by 119
  28. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  29. 2022MetaMorph: Learning Universal Controllers with Transformerscited by 89
  30. 2020Transformers for One-Shot Visual Imitationcited by 78
  31. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 71
  32. 2020My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Controlcited by 70
  33. 2020Imitating Interactive Intelligencecited by 55
  34. 2020Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigationcited by 32
  35. 2023Lossless Adaptation of Pretrained Vision Models For Robotic Manipulationcited by 29
  36. 2019Learning to Reach Goals via Iterated Supervised Learningcited by 26
  37. 2020Rapid Task-Solving in Novel Environmentscited by 23
  38. 2021Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulationcited by 13

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 27; 2025: 30; 2026: 50.

111 papers.

Sort:

By area:

  1. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  2. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 346
  3. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  4. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  5. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  6. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  7. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 142
  8. 2024A Survey on Self-Evolution of Large Language Modelscited by 91
  9. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 88
  10. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  11. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 59
  12. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 55
38 more
  1. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 55
  2. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  3. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 52
  4. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  5. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 43
  6. 2024JUICER: Data-Efficient Imitation Learning for Robotic Assemblycited by 38
  7. 2024Multi-Task Interactive Robot Fleet Learning with Visual World Modelscited by 36
  8. 2024Cross-Embodiment Dexterous Grasping with Reinforcement Learningcited by 27
  9. 2025AnyPos: Automated Task-Agnostic Actions for Bimanual Manipulationcited by 23
  10. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 20
  11. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 20
  12. 2024Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theorycited by 19
  13. 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interactioncited by 19
  14. 2024Natural Language Reinforcement Learningcited by 17
  15. 2024So You Think You Can Scale Up Autonomous Robot Data Collection?cited by 17
  16. 2026Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learningcited by 17
  17. 2025What Matters for Batch Online Reinforcement Learning in Robotics?cited by 15
  18. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 15
  19. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 14
  20. 2026Turning Video Models into Generalist Robot Policiescited by 12
  21. 2024DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robotscited by 11
  22. 2025EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Taskscited by 10
  23. 2025OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learningcited by 10
  24. 2026Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesiscited by 10
  25. 2024Active Fine-Tuning of Multi-Task Policiescited by 9
  26. 2025Co-Evolving Agents: Learning from Failures as Hard Negativescited by 8
  27. 2024Transforming Agency. On the mode of existence of Large Language Modelscited by 7
  28. 2025SIME: Enhancing Policy Self-Improvement with Modal-level Explorationcited by 7
  29. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  30. 2025Multi-Modal Multi-Task (M3T) Federated Foundation Models for Embodied AI: Potentials and Challenges for Edge Integrationcited by 6
  31. 2025AnyBody: A Benchmark Suite for Cross-Embodiment Manipulationcited by 6
  32. 2025SOE: Sample-Efficient Robot Policy Self-Improvement via On-Manifold Explorationcited by 6
  33. 2025DropVLA: An Action-Level Backdoor Attack on Vision-Language-Action Modelscited by 6
  34. 2026FocusVLA: Focused Visual Utilization for Vision-Language-Action Modelscited by 6
  35. 2026Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Trainingcited by 6
  36. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
  37. 2024Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learningcited by 5
  38. 2024Natural Language Reinforcement Learningcited by 5

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
  2. 2025RoBridge: A Hierarchical Architecture Bridging Cognition and Execution for General Robotic Manipulation
  3. 2021Towards More Generalizable One-shot Visual Imitation Learning
  4. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation
  5. 2026RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generation
  6. 2025RoboBERT: An End-to-end Multimodal Robotic Manipulation Model
  7. 2026Explore, Execute, Evolve: A Skill Acquisition and Reuse Loop for Embodied Agents
  8. 2024Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations