# Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learning

arXiv: https://arxiv.org/abs/2503.04877 · HTML: https://mlsanity.com/paper/2503.04877/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 58 of its 74 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)

## Built on by (most cited first, 17 of 17)

- [GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Model](https://mlsanity.com/paper/2509.14117/) (2025)
- [GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulation](https://mlsanity.com/paper/2509.15733/) (2025)
- [VLA Models Are More Generalizable Than You Think: Revisiting Physical and Spatial Modeling](https://mlsanity.com/paper/2512.02902/) (2025)
- [Geometric Action Model for Robot Policy Learning](https://mlsanity.com/paper/2606.17046/) (2026)
- [AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models](https://mlsanity.com/paper/2603.05868/) (2026)
- [HeRO: Hierarchical 3D Semantic Representation for Pose-aware Object Manipulation](https://mlsanity.com/paper/2602.18817/) (2026)
- [Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasets](https://mlsanity.com/paper/2601.09605/) (2026)
- [Fourier Features Let Agents Learn High Precision Policies with Imitation Learning](https://mlsanity.com/paper/2606.12334/) (2026)
- [Large Pre-Trained Models for Bimanual Manipulation in 3D](https://mlsanity.com/paper/2509.20579/) (2025)
- [Fiducial Exoskeletons: Image-Centric Robot State Estimation](https://mlsanity.com/paper/2601.08034/) (2026)
- [See like a Robot: Robot-Centric Pointmaps for VLA Models](https://mlsanity.com/paper/2607.11498/) (2026)
- [Point Cloud Sequence Encoding for Material-conditioned Graph Network Simulators](https://mlsanity.com/paper/2605.20978/) (2026)
- [Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shifts](https://mlsanity.com/paper/2607.00666/) (2026)
- [Learning 3D Affordances for Blade Insertion in Cluttered Stowing](https://mlsanity.com/paper/2607.02549/) (2026)
- [RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning](https://mlsanity.com/paper/2607.29622/) (2026)
- [Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis](https://mlsanity.com/paper/2610.03717/) (2026)
- [VersaCamVLA: Camera-Configurable VLA Policies for Robotic Manipulation](https://mlsanity.com/paper/2610.12451/) (2026)

## Abstract (arXiv)

Imitation Learning can train robots to perform complex and diverse manipulation tasks, but learned policies are brittle with observations outside of the training distribution. 3D scene representations that incorporate observations from calibrated RGBD cameras have been proposed as a way to mitigate this, but in our evaluations with unseen embodiments and camera viewpoints they show only modest improvement. To address those challenges, we propose Adapt3R, a general-purpose 3D observation encoder which synthesizes data from calibrated RGBD cameras into a vector that can be used as conditioning for arbitrary IL algorithms. The key idea is to use a pretrained 2D backbone to extract semantic information, using 3D only as a medium to localize this information with respect to the end-effector. We show across 93 simulated and 6 real tasks that when trained end-to-end with a variety of IL algorithms, Adapt3R maintains these algorithms' learning capacity while enabling zero-shot transfer to novel embodiments and camera poses.