mlsanityData to 7 Oct 2026

Scaling Robot Learning with Semantically Imagined Experience

arXiv 2302.11550 ↗First posted 22 Feb 2023cs.RO cs.AI cs.CLcited by 175

Not on the map. Why?

Cited by 34 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in robot learning have shown promise in enabling robots to perform a variety of manipulation tasks and generalize to novel scenarios. One of the key contributing factors to this progress is the scale of robot data used to train the models. To obtain large-scale datasets, prior approaches have relied on either demonstrations requiring high human involvement or engineering-heavy autonomous data collection schemes, both of which are challenging to scale. To mitigate this issue, we propose an alternative route and leverage text-to-image foundation models widely used in computer vision and natural language processing to obtain meaningful data for robot learning without requiring additional robot data. We term our method Robot Learning with Semantically Imagened Experience (ROSIE). Specifically, we make use of the state of the art text-to-image diffusion models and perform aggressive data augmentation on top of our existing robotic manipulation datasets via inpainting various unseen objects for manipulation, backgrounds, and distractors with text guidance. Through extensive real-world experiments, we show that manipulation policies trained on data augmented this way are able to solve completely unseen tasks with new objects and can behave more robustly w.r.t. novel distractors. In addition, we find that we can improve the robustness and generalization of high-level robot learning tasks such as success detection through training with the diffusion-based data augmentation. The project's website and videos can be found at diffusion-rosie.github.io

Key facts. Cited by 175 papers in the map (data through 7 Oct 2026). Builds on 69 of its 82 references that are in the map. First posted on arXiv 22 Feb 2023.

Builds on

69 of 82 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2017Mask R-CNNcited by 7,259
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  13. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  14. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  15. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  16. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  17. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  18. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  19. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  20. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  21. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  22. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  23. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  24. 2019Solving Rubik's Cube with a Robot Handcited by 699
  25. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  26. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  27. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  28. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  29. 2018Generative Image Inpainting with Contextual Attentioncited by 578
  30. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  31. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  32. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  33. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  34. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  35. 2020Reinforcement Learning with Augmented Datacited by 452
  36. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  37. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  38. 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 292

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 34; 2024: 59; 2025: 50; 2026: 32.

175 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  4. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  5. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  6. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  7. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  8. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  9. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  10. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  11. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  12. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
38 more
  1. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 120
  2. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  3. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 106
  4. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 104
  5. 2023Synthetic Experience Replaycited by 100
  6. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  7. 2023Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Modelscited by 87
  8. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  9. 2024Video as the New Language for Real-World Decision Makingcited by 86
  10. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  11. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  12. 2024EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learningcited by 72
  13. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  14. 2025Humanoid Policy ~ Human Policycited by 65
  15. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  16. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 60
  17. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  18. 2023Language Models as Zero-Shot Trajectory Generatorscited by 59
  19. 2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
  20. 2025OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interactioncited by 57
  21. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  22. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
  23. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  24. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  25. 2025Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policiescited by 49
  26. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 48
  27. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  28. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  29. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 40
  30. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  31. 2024Instant Policy: In-Context Imitation Learning via Graph Diffusioncited by 38
  32. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37
  33. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  34. 2023EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulationcited by 37
  35. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  36. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  37. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 36
  38. 2024Grounding Multimodal Large Language Models in Actionscited by 35

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025S2-Diffusion: Generalizing from Instance-level to Category-level Skills in Robot Manipulation
  2. 2025LaVA-Man: Learning Visual Action Representations for Robot Manipulation
  3. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulation
  4. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation
  5. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructions
  6. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  7. 2026RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation
  8. 2025RoboEngine: Plug-and-Play Robot Data Augmentation with Semantic Robot Segmentation and Background Generation