mlsanityData to 14 Oct 2026

Learning Generalizable Manipulation Policies with Object-Centric 3D Representations

arXiv 2310.14386 ↗First posted 22 Oct 2023cs.RO cs.CV cs.LGcited by 61

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We introduce GROOT, an imitation learning method for learning robust policies with object-centric and 3D priors. GROOT builds policies that generalize beyond their initial training conditions for vision-based manipulation. It constructs object-centric 3D representations that are robust toward background changes and camera views and reason over these representations using a transformer-based policy. Furthermore, we introduce a segmentation correspondence model that allows policies to generalize to new objects at test time. Through comprehensive experiments, we validate the robustness of GROOT policies against perceptual variations in simulated and real-world environments. GROOT's performance excels in generalization over background changes, camera viewpoint shifts, and the presence of new object instances, whereas both state-of-the-art end-to-end learning methods and object proposal-based approaches fall short. We also extensively evaluate GROOT policies on real robots, where we demonstrate the efficacy under very wild changes in setup. More videos and model details can be found in the appendix and the project website: https://ut-austin-rpl.github.io/GROOT .

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 54 of its 61 references that are in the map. First posted on arXiv 22 Oct 2023.

Builds on

54 of 61 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play24 Feb
  2. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning12 Dec
  3. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning20 Oct
  4. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors20 Oct
  5. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  6. 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control28 Feb
  7. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations13 Mar
  8. 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep

Other references · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Segment Anythingcited by 8,234
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  7. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  8. 2020Point Transformercited by 1,470
  9. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  10. 2020Object-Centric Learning with Slot Attentioncited by 876
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  12. 2018Open3D: A Modern Library for 3D Data Processingcited by 721
34 more
  1. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  2. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  3. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  4. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  5. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  6. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  7. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  8. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  9. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  10. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 289
  11. 2019Causal Confusion in Imitation Learningcited by 288
  12. 2022Masked Visual Pre-training for Motor Controlcited by 248
  13. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 241
  14. 2017One-Shot Imitation Learningcited by 233
  15. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  16. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  17. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 156
  18. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 153
  19. 2021Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusioncited by 129
  20. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 108
  21. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 104
  22. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  23. 20226-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmarkcited by 86
  24. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 80
  25. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 71
  26. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 70
  27. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 56
  28. 2017Deep Object-Centric Representations for Generalizable Robot Learningcited by 50
  29. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 47
  30. 2019Graph-Structured Visual Imitationcited by 43
  31. 2018Deep Object-Centric Policies for Autonomous Drivingcited by 41
  32. 2019Object-Centric Task and Motion Planning in Dynamic Environmentscited by 35
  33. 2022Visuomotor Control in Multi-Object Scenes Using Object-Aware Representationscited by 21
  34. 2021Object-Aware Regularization for Addressing Causal Confusion in Imitation Learningcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 19; 2025: 21; 2026: 19.

61 papers.

Sort:

By area:

  1. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  2. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  3. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 145
  4. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 133
  5. 2023LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discoverycited by 67
  6. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 67
  7. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 54
  8. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 50
  9. 2025ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Modelscited by 43
  10. 2024AffordDP: Generalizable Diffusion Policy with Transferable Affordancecited by 37
  11. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  12. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
38 more
  1. 2024Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robustcited by 33
  2. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 30
  3. 2025GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Modelcited by 26
  4. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  5. 2025FlightGPT: Towards Generalizable and Interpretable UAV Vision-and-Language Navigation with Vision-Language Modelscited by 22
  6. 2023Harmonic Mobile Manipulationcited by 17
  7. 2024Object-Centric Instruction Augmentation for Robotic Manipulationcited by 16
  8. 2024Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Modelscited by 16
  9. 2024P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policiescited by 16
  10. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 14
  11. 2024Language-Guided Object-Centric Diffusion Policy for Generalizable and Collision-Aware Robotic Manipulationcited by 14
  12. 2025Object-centric 3D Motion Field for Robot Learning from Human Videoscited by 14
  13. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 14
  14. 2025Mobi-π: Mobilizing Your Robot Learning Policycited by 10
  15. 2025Tool-as-Interface: Learning Robot Policies from Observing Human Tool Usecited by 8
  16. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguitycited by 8
  17. 2024Recasting Generic Pretrained Vision Transformers As Object-Centric Scene Encoders For Manipulation Policiescited by 6
  18. 2026OCRA: Object-Centric Learning with 3D and Tactile Priors for Human-to-Robot Action Transfercited by 6
  19. 2025Compose by Focus: Scene Graph-based Atomic Skillscited by 4
  20. 2025GAE: Unleashing Physical Potential of VLM with Generalizable Action Expertcited by 4
  21. 2025PosA-VLA: Enhancing Action Generation via Pose-Conditioned Anchor Attentioncited by 4
  22. 2025CIVIL: Causal and Intuitive Visual Imitation Learningcited by 3
  23. 2025Improving Robotic Manipulation Robustness via NICE Scene Surgerycited by 3
  24. 2025Enhancing Reusability of Learned Skills for Robot Manipulation via Gaze Information and Motion Bottleneckscited by 2
  25. 2025Learning Predictive Visuomotor Coordinationcited by 2
  26. 2025Generalizable Hierarchical Skill Learning via Object-Centric Representationcited by 2
  27. 2026Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasetscited by 2
  28. 2026On the Generalization Capabilities, Design Choices and Limitations of Keypoint Imitation Learningcited by 2
  29. 2025VLM-TDP: VLM-guided Trajectory-conditioned Diffusion Policy for Robust Long-Horizon Manipulationcited by 1
  30. 2025Intuitive Programming, Adaptive Task Planning, and Dynamic Role Allocation in Human-Robot Collaborationcited by 1
  31. 2026CUBic: Coordinated Unified Bimanual Perception and Control Frameworkcited by 1
  32. 20263D-DLP: Self-Supervised 3D Object-Centric Scene Representation Learningcited by 1
  33. 2026From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligencecited by 1
  34. 2024Learning Generalizable 3D Manipulation With 10 Demonstrationscited by 0
  35. 2025Object-Focus Actor for Data-efficient Robot Generalization Dexterous Manipulationcited by 0
  36. 2025GLUE: Global-Local Unified Encoding for Imitation Learning via Key-Patch Trackingcited by 0
  37. 2025Learning to Grasp Anything by Playing with Random Toyscited by 0
  38. 2026Learning Object-Centric Spatial Reasoning for Sequential Manipulation in Cluttered Environmentscited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Learning Category-Level Generalizable Object Manipulation Policy via Generative Adversarial Self-Imitation Learning from Demonstrations
  2. 2023Generalizable Imitation Learning Through Pre-Trained Representations
  3. 2026KeyGen: Unsupervised Keypoint based Object-Centric Representations for Category-Level Policy Generalization
  4. 2024Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Frames
  5. 2025Zero-Shot Visual Generalization in Robot Manipulation
  6. 2026An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning
  7. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations
  8. 2025PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies