mlsanityData to 7 Oct 2026

VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors

arXiv 2210.11339 ↗First posted 20 Oct 2022cs.ROcited by 135

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We introduce VIOLA, an object-centric imitation learning approach to learning closed-loop visuomotor policies for robot manipulation. Our approach constructs object-centric representations based on general object proposals from a pre-trained vision model. VIOLA uses a transformer-based policy to reason over these representations and attend to the task-relevant visual factors for action prediction. Such object-based structural priors improve deep imitation learning algorithm's robustness against object variations and environmental perturbations. We quantitatively evaluate VIOLA in simulation and on real robots. VIOLA outperforms the state-of-the-art imitation learning methods by 45.8% in success rate. It has also been deployed successfully on a physical robot to solve challenging long-horizon tasks, such as dining table arrangement and coffee making. More videos and model details can be found in supplementary material and the project website: https://ut-austin-rpl.github.io/VIOLA .

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 54 of its 67 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

54 of 67 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation28 Sep
  2. 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control28 Feb
  3. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations13 Mar
  4. 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2017Mask R-CNNcited by 7,259
  7. 2016Layer Normalizationcited by 5,579
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  10. 2017Random Erasing Data Augmentationcited by 1,235
  11. 2019Objects as Pointscited by 1,172
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
38 more
  1. 2020Object-Centric Learning with Slot Attentioncited by 865
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  3. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  4. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  5. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  6. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  7. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  8. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  9. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  10. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  11. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  12. 2019Causal Confusion in Imitation Learningcited by 286
  13. 2020Critic Regularized Regressioncited by 269
  14. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 237
  15. 2020On the Binding Problem in Artificial Neural Networkscited by 230
  16. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  17. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  18. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  19. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  20. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
  21. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  22. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 107
  23. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  24. 2021Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Cluttercited by 94
  25. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  26. 20226-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmarkcited by 85
  27. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  28. 2020Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networkscited by 77
  29. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 70
  30. 2017Deep Object-Centric Representations for Generalizable Robot Learningcited by 50
  31. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  32. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  33. 2019Graph-Structured Visual Imitationcited by 43
  34. 2018Deep Object-Centric Policies for Autonomous Drivingcited by 41
  35. 2019Object-Centric Task and Motion Planning in Dynamic Environmentscited by 34
  36. 2020The MAGICAL Benchmark for Robust Imitationcited by 30
  37. 2021Object-Aware Regularization for Addressing Causal Confusion in Imitation Learningcited by 16
  38. 2020Learning to Play by Imitating Humanscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 40; 2025: 49; 2026: 23.

135 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  4. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  5. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  6. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  7. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  8. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  9. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  10. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  11. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  12. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
38 more
  1. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  2. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  3. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  4. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  5. 2025Unified Vision-Language-Action Modelcited by 128
  6. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  7. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  8. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  9. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  10. 2023Data Quality in Imitation Learningcited by 72
  11. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  12. 2023LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discoverycited by 63
  13. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 62
  14. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
  15. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 60
  16. 2023Waypoint-Based Imitation Learning for Robotic Manipulationcited by 59
  17. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  18. 2023HYDRA: Hybrid Robot Actions for Imitation Learningcited by 54
  19. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 51
  20. 2024AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policiescited by 50
  21. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  22. 2023Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperationcited by 49
  23. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  24. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  25. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
  26. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 43
  27. 2025ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Modelscited by 43
  28. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
  29. 2024Diffusion Transformer Policycited by 41
  30. 2024Learning Manipulation by Predicting Interactioncited by 36
  31. 2025Failure Prediction at Runtime for Generative Robot Policiescited by 36
  32. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  33. 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
  34. 2023MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulationcited by 30
  35. 2023Human-in-the-Loop Task and Motion Planning for Imitation Learningcited by 28
  36. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 28
  37. 2023MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulationcited by 27
  38. 2024ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matchingcited by 25

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning
  2. 2025ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstration
  3. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulation
  4. 2026ICLR: In-Context Imitation Learning with Visual Reasoning
  5. 2025Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions
  6. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
  7. 2024DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models
  8. 2026GAP: Geometric Anchor Pre-training for Data-Efficient Visuomotor Learning of Manipulation Tasks