# Visual Reinforcement Learning with Self-Supervised 3D Representations

arXiv: https://arxiv.org/abs/2210.07241 · HTML: https://mlsanity.com/paper/2210.07241/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 80 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 67)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [RoboNet: Large-Scale Multi-Robot Learning](https://mlsanity.com/paper/1910.11215/) (2019)
- [Decoupling Representation Learning from Reinforcement Learning](https://mlsanity.com/paper/2009.08319/) (2020)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Masked Visual Pre-training for Motor Control](https://mlsanity.com/paper/2203.06173/) (2022)
- [Real-World Robot Learning with Masked Visual Pre-training](https://mlsanity.com/paper/2210.03109/) (2022)

## Built on by (most cited first, 30 of 45)

- [TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12514/) (2024)
- [3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations](https://mlsanity.com/paper/2403.03954/) (2024)
- [ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation](https://mlsanity.com/paper/2403.08321/) (2024)
- [GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields](https://mlsanity.com/paper/2308.16891/) (2023)
- [RayZer: A Self-supervised Large View Synthesis Model](https://mlsanity.com/paper/2505.00702/) (2025)
- [WorldEval: World Model as Real-World Robot Policies Evaluator](https://mlsanity.com/paper/2505.19017/) (2025)
- [On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline](https://mlsanity.com/paper/2212.05749/) (2022)
- [ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledge](https://mlsanity.com/paper/2505.21906/) (2025)
- [Finetuning Offline World Models in the Real World](https://mlsanity.com/paper/2310.16029/) (2023)
- [Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning](https://mlsanity.com/paper/2407.15815/) (2024)
- [Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation](https://mlsanity.com/paper/2409.14411/) (2024)
- [D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement](https://mlsanity.com/paper/2309.16118/) (2023)
- [CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation](https://mlsanity.com/paper/2402.14795/) (2024)
- [Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization](https://mlsanity.com/paper/2311.03351/) (2023)
- [Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning](https://mlsanity.com/paper/2310.20587/) (2023)
- [DNAct: Diffusion Guided Multi-Task 3D Policy Learning](https://mlsanity.com/paper/2403.04115/) (2024)
- [Continuous Control with Coarse-to-fine Reinforcement Learning](https://mlsanity.com/paper/2407.07787/) (2024)
- [MoVie: Visual Model-Based Policy Adaptation for View Generalization](https://mlsanity.com/paper/2307.00972/) (2023)
- [On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning](https://mlsanity.com/paper/2210.10763/) (2022)
- [Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer](https://mlsanity.com/paper/2512.01061/) (2025)
- [Point Cloud Models Improve Visual Robustness in Robotic Learners](https://mlsanity.com/paper/2404.18926/) (2024)
- [Reinforcement Learning with Generalizable Gaussian Splatting](https://mlsanity.com/paper/2404.07950/) (2024)
- [Generalizable Visual Reinforcement Learning with Segment Anything Model](https://mlsanity.com/paper/2312.17116/) (2023)
- [PointCaM: Cut-and-Mix for Open-Set Point Cloud Learning](https://mlsanity.com/paper/2212.02011/) (2022)
- [On the Efficacy of 3D Point Cloud Reinforcement Learning](https://mlsanity.com/paper/2306.06799/) (2023)
- [ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model](https://mlsanity.com/paper/2506.19842/) (2025)
- [Policy Adaptation from Foundation Model Feedback](https://mlsanity.com/paper/2212.07398/) (2022)
- [Robo360: A 3D Omnispective Multi-Material Robotic Manipulation Dataset](https://mlsanity.com/paper/2312.06686/) (2023)
- [4D Visual Pre-training for Robot Learning](https://mlsanity.com/paper/2508.17230/) (2025)
- [Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence](https://mlsanity.com/paper/2402.02701/) (2024)

## Abstract (arXiv)

A prominent approach to visual Reinforcement Learning (RL) is to learn an internal state representation using self-supervised methods, which has the potential benefit of improved sample-efficiency and generalization through additional learning signal and inductive biases. However, while the real world is inherently 3D, prior efforts have largely been focused on leveraging 2D computer vision techniques as auxiliary self-supervision. In this work, we present a unified framework for self-supervised learning of 3D representations for motor control. Our proposed framework consists of two phases: a pretraining phase where a deep voxel-based 3D autoencoder is pretrained on a large object-centric dataset, and a finetuning phase where the representation is jointly finetuned together with RL on in-domain data. We empirically show that our method enjoys improved sample efficiency in simulated manipulation tasks compared to 2D representation learning methods. Additionally, our learned policies transfer zero-shot to a real robot setup with only approximate geometric correspondence, and successfully solve motor control tasks that involve grasping and lifting from a single, uncalibrated RGB camera. Code and videos are available at https://yanjieze.com/3d4rl/ .