mlsanityData to 7 Oct 2026

RVT: Robotic View Transformer for 3D Object Manipulation

arXiv 2306.14896 ↗First posted 26 Jun 2023cs.RO cs.CVcited by 224

Not on the map. Why?

Cited by 55 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

For 3D object manipulation, methods that build an explicit 3D representation perform better than those relying only on camera images. But using explicit 3D representations like voxels comes at large computing cost, adversely affecting scalability. In this work, we propose RVT, a multi-view transformer for 3D manipulation that is both scalable and accurate. Some key features of RVT are an attention mechanism to aggregate information across views and re-rendering of the camera input from virtual views around the robot workspace. In simulations, we find that a single RVT model works well across 18 RLBench tasks with 249 task variations, achieving 26% higher relative success than the existing state-of-the-art method (PerAct). It also trains 36X faster than PerAct for achieving the same performance and achieves 2.3X the inference speed of PerAct. Further, RVT can perform a variety of manipulation tasks in the real world with just a few (~10) demonstrations per task. Visual results, code, and trained model are provided at https://robotic-view-transformer.github.io/.

Key facts. Cited by 224 papers in the map (data through 7 Oct 2026). Builds on 56 of its 69 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

56 of 69 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  11. 2022A Generalist Agentcited by 802
  12. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
38 more
  1. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  2. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  3. 2018Learning Dexterous In-Hand Manipulationcited by 643
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  8. 2018On Evaluation of Embodied Navigation Agentscited by 526
  9. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  10. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  11. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  13. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  14. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  15. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  16. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  17. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  18. 2021A System for General In-Hand Object Re-Orientationcited by 155
  19. 2021Revisiting Point Cloud Shape Classification with a Simple and Effective Baselinecited by 145
  20. 2020MVTN: Multi-View Transformation Network for 3D Shape Recognitioncited by 140
  21. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  22. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  23. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 124
  24. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  25. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  26. 2013Deep Learning for Detecting Robotic Graspscited by 114
  27. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  28. 2020Transformers for One-Shot Visual Imitationcited by 77
  29. 2020ViNG: Learning Open-World Navigation with Visual Goalscited by 77
  30. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  31. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  32. 2021Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learningcited by 49
  33. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  34. 2022IFOR: Iterative Flow Minimization for Robotic Object Rearrangementcited by 42
  35. 2021Transformer-based deep imitation learning for dual-arm robot manipulationcited by 31
  36. 2020Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3Dcited by 30
  37. 2022Offline Reinforcement Learning for Visual Navigationcited by 28
  38. 2022MIRA: Mental Imagery for Robotic Affordancescited by 26

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 66; 2025: 88; 2026: 69.

224 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  2. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  4. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  5. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  6. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  7. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  8. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  9. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  10. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  11. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  12. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
38 more
  1. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  2. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 101
  3. 2024Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulationcited by 92
  4. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  5. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  6. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  7. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
  8. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  9. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  10. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  11. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  12. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  13. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  14. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modificationcited by 49
  15. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  16. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 46
  17. 2024VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulationcited by 45
  18. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  19. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  20. 2025ManiFlow: A General Robot Manipulation Policy via Consistency Flow Trainingcited by 43
  21. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  22. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  23. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  24. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  25. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  26. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
  27. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  28. 2024Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3Dcited by 32
  29. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
  30. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  31. 2024BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmarkcited by 30
  32. 2025LLaDA-VLA: Vision Language Diffusion Action Modelscited by 30
  33. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
  34. 2025Pre-training Auto-regressive Robotic Models with 4D Representationscited by 27
  35. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  36. 2024Autoregressive Action Sequence Learning for Robotic Manipulationcited by 26
  37. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 26
  38. 2024ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matchingcited by 25

50 most cited of 224 All 224 (JSON)