mlsanityData to 14 Oct 2026

CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning

arXiv 2212.05711 ↗First posted 12 Dec 2022cs.RO cs.AI cs.LGcited by 71

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Large-scale training have propelled significant progress in various sub-fields of AI such as computer vision and natural language processing. However, building robot learning systems at a comparable scale remains challenging. To develop robots that can perform a wide range of skills and adapt to new scenarios, efficient methods for collecting vast and diverse amounts of data on physical robot systems are required, as well as the capability to train high-capacity policies using such datasets. In this work, we propose a framework for scaling robot learning, with specific focus on multi-task and multi-scene manipulation in kitchen environments, both in simulation and in the real world. Our proposed framework, CACTI, comprises four stages that separately handle: data collection, data augmentation, visual representation learning, and imitation policy training, to enable scalability in robot learning . We make use of state-of-the-art generative models as part of the data augmentation stage, and use pre-trained out-of-domain visual representations to improve training efficiency. Experimental results demonstrate the effectiveness of our approach. On a real robot setup, CACTI enables efficient training of a single policy that can perform 10 manipulation tasks involving kitchen objects, and is robust to varying layouts of distractors. In a simulated kitchen environment, CACTI trains a single policy to perform 18 semantic tasks across 100 layout variations for each individual task. We will release the simulation task benchmark and augmented datasets in both real and simulated environments to facilitate future research.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 59 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

45 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Behavior Transformers: Cloning k modes with one stone22 Jun
  2. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  3. 2020Visual Imitation Made Easy11 Aug
  4. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning25 Oct
  5. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation7 Nov

Other references · 40

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  6. 2018World Modelscited by 1,513
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  8. 2016Progressive Neural Networkscited by 1,361
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  11. 2018DeepMind Control Suitecited by 956
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
28 more
  1. 2022A Generalist Agentcited by 808
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  4. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  5. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  6. 2020Reinforcement Learning with Augmented Datacited by 468
  7. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  8. 2019Learning Latent Plans from Playcited by 316
  9. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  10. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  11. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  12. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  13. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
  14. 2017Stochastic Variational Video Predictioncited by 234
  15. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 233
  16. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  17. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  18. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  19. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 174
  20. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  21. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  22. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  23. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 106
  24. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  25. 2019Shaping Belief States with Generative Environment Models for RLcited by 66
  26. 2016Learning to Push by Grasping: Using multiple tasks for effective learningcited by 37
  27. 2015Adapting Deep Visuomotor Representations with Weak Pairwise Constraintscited by 25
  28. 2020Latent Skill Planning for Exploration and Transfercited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 26; 2025: 22; 2026: 9.

71 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  4. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 339
  5. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
  6. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 282
  7. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  8. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  9. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  10. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 169
  11. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  12. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
38 more
  1. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 144
  2. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
  3. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
  4. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 116
  5. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 111
  6. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  7. 2025OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interactioncited by 71
  8. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 70
  9. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 61
  10. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  11. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 52
  12. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 52
  13. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  14. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  15. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 35
  16. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
  17. 2024Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robustcited by 33
  18. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 28
  19. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  20. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 23
  21. 2025ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesiscited by 22
  22. 2025Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimizationcited by 21
  23. 2026SOP: A Scalable Online Post-Training System for Vision-Language-Action Modelscited by 21
  24. 2024Render and Diffuse: Aligning Image and Action Spaces for Diffusion-based Behaviour Cloningcited by 20
  25. 2023Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMscited by 19
  26. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 19
  27. 2024Semantically Controllable Augmentations for Generalizable Robot Learningcited by 18
  28. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  29. 2023Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptationcited by 16
  30. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  31. 2024A Survey of Imitation Learning Methods, Environments and Metricscited by 16
  32. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 15
  33. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 14
  34. 2025ManipDreamer3D : Synthesizing Plausible Robotic Manipulation Video with Occupancy-aware 3D Trajectorycited by 12
  35. 2025EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfercited by 12
  36. 2024Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Controlcited by 11
  37. 2024Green Screen Augmentation Enables Scene Generalisation in Robotic Manipulationcited by 11
  38. 2025OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learningcited by 10

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning
  2. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation
  3. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulation
  4. 2021Towards More Generalizable One-shot Visual Imitation Learning
  5. 2025Zero-Shot Visual Generalization in Robot Manipulation
  6. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
  7. 2026Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassembly
  8. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot