# RVT: Robotic View Transformer for 3D Object Manipulation

arXiv: https://arxiv.org/abs/2306.14896 · HTML: https://mlsanity.com/paper/2306.14896/

## Key facts

- Cited by 225 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 69 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)
- [A System for General In-Hand Object Re-Orientation](https://mlsanity.com/paper/2111.03043/) (2021)

## Built on by (most cited first, 30 of 225)

- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics](https://mlsanity.com/paper/2406.10721/) (2024)
- [MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2508.19236/) (2025)
- [HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model](https://mlsanity.com/paper/2503.10631/) (2025)
- [RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation](https://mlsanity.com/paper/2412.13877/) (2024)
- [Agent AI: Surveying the Horizons of Multimodal Interaction](https://mlsanity.com/paper/2401.03568/) (2024)
- [RVT-2: Learning Precise Manipulation from Few Demonstrations](https://mlsanity.com/paper/2406.08545/) (2024)
- [THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation](https://mlsanity.com/paper/2402.08191/) (2024)
- [HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation](https://mlsanity.com/paper/2502.05485/) (2025)
- [3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations](https://mlsanity.com/paper/2403.03954/) (2024)
- [ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation](https://mlsanity.com/paper/2403.08321/) (2024)
- [PointVLA: Injecting the 3D World into Vision-Language-Action Models](https://mlsanity.com/paper/2503.07511/) (2025)
- [Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2403.03890/) (2024)
- [Manipulate-Anything: Automating Real-World Robots using Vision-Language Models](https://mlsanity.com/paper/2406.18915/) (2024)
- [BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models](https://mlsanity.com/paper/2506.07961/) (2025)
- [Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning](https://mlsanity.com/paper/2407.03162/) (2024)
- [Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation](https://mlsanity.com/paper/2410.08001/) (2024)
- [Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress](https://mlsanity.com/paper/2410.04640/) (2024)
- [Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning](https://mlsanity.com/paper/2501.06994/) (2025)
- [PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators](https://mlsanity.com/paper/2406.20083/) (2024)
- [Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learning](https://mlsanity.com/paper/2407.01531/) (2024)
- [AffordanceLLM: Grounding Affordance from Vision Language Models](https://mlsanity.com/paper/2401.06341/) (2024)
- [Affordance-based Robot Manipulation with Flow Matching](https://mlsanity.com/paper/2409.01083/) (2024)
- [SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment](https://mlsanity.com/paper/2410.18907/) (2024)
- [RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning](https://mlsanity.com/paper/2409.14674/) (2024)
- [ManiFlow: A General Robot Manipulation Policy via Consistency Flow Training](https://mlsanity.com/paper/2509.01819/) (2025)
- [VLA-0: Building State-of-the-Art VLAs with Zero Modification](https://mlsanity.com/paper/2510.13054/) (2025)
- [VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation](https://mlsanity.com/paper/2411.09153/) (2024)

## Abstract (arXiv)

For 3D object manipulation, methods that build an explicit 3D representation perform better than those relying only on camera images. But using explicit 3D representations like voxels comes at large computing cost, adversely affecting scalability. In this work, we propose RVT, a multi-view transformer for 3D manipulation that is both scalable and accurate. Some key features of RVT are an attention mechanism to aggregate information across views and re-rendering of the camera input from virtual views around the robot workspace. In simulations, we find that a single RVT model works well across 18 RLBench tasks with 249 task variations, achieving 26% higher relative success than the existing state-of-the-art method (PerAct). It also trains 36X faster than PerAct for achieving the same performance and achieves 2.3X the inference speed of PerAct. Further, RVT can perform a variety of manipulation tasks in the real world with just a few (~10) demonstrations per task. Visual results, code, and trained model are provided at https://robotic-view-transformer.github.io/.