mlsanityData to 7 Oct 2026

Visual Reinforcement Learning with Self-Supervised 3D Representations

arXiv 2210.07241 ↗First posted 13 Oct 2022cs.LG cs.ROcited by 45

Not on the map. Why?

Abstract (arXiv)

A prominent approach to visual Reinforcement Learning (RL) is to learn an internal state representation using self-supervised methods, which has the potential benefit of improved sample-efficiency and generalization through additional learning signal and inductive biases. However, while the real world is inherently 3D, prior efforts have largely been focused on leveraging 2D computer vision techniques as auxiliary self-supervision. In this work, we present a unified framework for self-supervised learning of 3D representations for motor control. Our proposed framework consists of two phases: a pretraining phase where a deep voxel-based 3D autoencoder is pretrained on a large object-centric dataset, and a finetuning phase where the representation is jointly finetuned together with RL on in-domain data. We empirically show that our method enjoys improved sample efficiency in simulated manipulation tasks compared to 2D representation learning methods. Additionally, our learned policies transfer zero-shot to a real robot setup with only approximate geometric correspondence, and successfully solve motor control tasks that involve grasping and lifting from a single, uncalibrated RGB camera. Code and videos are available at https://yanjieze.com/3d4rl/ .

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 67 of its 80 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

67 of 80 references.

References · 67

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2020Object-Centric Learning with Slot Attentioncited by 865
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
38 more
  1. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  4. 2018Learning Dexterous In-Hand Manipulationcited by 643
  5. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  8. 2020Reinforcement Learning with Augmented Datacited by 452
  9. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  10. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  11. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  12. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  13. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  14. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  15. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  16. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  17. 2022Masked Visual Pre-training for Motor Controlcited by 246
  18. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  19. 2021Mastering Atari Games with Limited Datacited by 235
  20. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  21. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  22. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  23. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  24. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  25. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  26. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  27. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  28. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  29. 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
  30. 2021Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objectscited by 119
  31. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  32. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  33. 2018Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Networkcited by 109
  34. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  35. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  36. 2019Self-supervised 6D Object Pose Estimation for Robot Manipulationcited by 100
  37. 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
  38. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 13; 2024: 13; 2025: 14; 2026: 1.

45 papers.

Sort:

By area:

  1. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  2. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  3. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  4. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  5. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 74
  6. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  7. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  8. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  9. 2023Finetuning Offline World Models in the Real Worldcited by 39
  10. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  11. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  12. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
33 more
  1. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  2. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  3. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  4. 2024DNAct: Diffusion Guided Multi-Task 3D Policy Learningcited by 22
  5. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  6. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  7. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  8. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 13
  9. 2025Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfercited by 13
  10. 2023Generalizable Visual Reinforcement Learning with Segment Anything Modelcited by 12
  11. 2022PointCaM: Cut-and-Mix for Open-Set Point Cloud Learningcited by 11
  12. 2024Reinforcement Learning with Generalizable Gaussian Splattingcited by 11
  13. 2022Policy Adaptation from Foundation Model Feedbackcited by 10
  14. 2025ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Modelcited by 10
  15. 2023On the Efficacy of 3D Point Cloud Reinforcement Learningcited by 9
  16. 2023Robo360: A 3D Omnispective Multi-Material Robotic Manipulation Datasetcited by 8
  17. 20254D Visual Pre-training for Robot Learningcited by 8
  18. 2024Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidencecited by 5
  19. 2025ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representationscited by 5
  20. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  21. 2025Scalable Multi-Task Reinforcement Learning for Generalizable Spatial Intelligence in Visuomotor Agentscited by 3
  22. 2024iGAiVA: Integrated Generative AI and Visual Analytics in a Machine Learning Workflow for Text Classificationcited by 2
  23. 2025Sample from What You See: Visuomotor Policy Learning via Diffusion Bridge with Observation-Embedded Stochastic Differential Equationcited by 2
  24. 2023H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulationcited by 1
  25. 2025Learning Fused State Representations for Control from Multi-View Observationscited by 1
  26. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Datacited by 1
  27. 2023Model-Based Reinforcement Learning with Multi-Task Offline Pretrainingcited by 0
  28. 2023Self-supervised Learning for Anomaly Detection in Computational Workflowscited by 0
  29. 2024Learning Collective Dynamics of Multi-Agent Systems using Event-based Visioncited by 0
  30. 2025Multimodal Fusion for Sim2real Transfer in Visual Reinforcement Learningcited by 0
  31. 2025Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learningcited by 0
  32. 2025ISS Policy : Scalable Diffusion Policy with Implicit Scene Supervisioncited by 0
  33. 2026Temporal Action Representation Learning for Tactical Resource Control and Subsequent Maneuver Generationcited by 0