mlsanityData to 7 Oct 2026

Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

arXiv 2209.05451 ↗First posted 12 Sep 2022cs.RO cs.AI cs.CLcited by 545

Not on the map. Why?

Cited by 93 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers have revolutionized vision and natural language processing with their ability to scale with large datasets. But in robotic manipulation, data is both limited and expensive. Can manipulation still benefit from Transformers with the right problem formulation? We investigate this question with PerAct, a language-conditioned behavior-cloning agent for multi-task 6-DoF manipulation. PerAct encodes language goals and RGB-D voxel observations with a Perceiver Transformer, and outputs discretized actions by ``detecting the next best voxel action''. Unlike frameworks that operate on 2D images, the voxelized 3D observation and action space provides a strong structural prior for efficiently learning 6-DoF actions. With this formulation, we train a single multi-task Transformer for 18 RLBench tasks (with 249 variations) and 7 real-world tasks (with 18 variations) from just a few demonstrations per task. Our results show that PerAct significantly outperforms unstructured image-to-action agents and 3D ConvNet baselines for a wide range of tabletop tasks.

Key facts. Cited by 545 papers in the map (data through 7 Oct 2026). Builds on 89 of its 116 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

89 of 116 references.

References · 89

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Mask R-CNNcited by 7,259
  12. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  3. 2021Vision Transformers for Dense Predictioncited by 1,789
  4. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  10. 2020Object-Centric Learning with Slot Attentioncited by 865
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  13. 2022A Generalist Agentcited by 802
  14. 2020Efficient Transformers: A Surveycited by 784
  15. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  16. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  17. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  18. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  19. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  20. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  21. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  22. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  23. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  24. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  25. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  26. 2021Implicit Behavioral Cloningcited by 442
  27. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  28. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  29. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  30. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  31. 2020Integrated Task and Motion Planningcited by 281
  32. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  33. 2021Voxel Transformer for 3D Object Detectioncited by 247
  34. 2020Experience Grounds Languagecited by 241
  35. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  36. 2022Multi-Game Decision Transformerscited by 212
  37. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  38. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 103; 2024: 149; 2025: 147; 2026: 138.

545 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  5. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  6. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  7. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  8. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 422
  9. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  10. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  11. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  12. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
38 more
  1. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  2. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  3. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  4. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  5. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  6. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  7. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  8. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  9. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  10. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  11. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  12. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  13. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  14. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  15. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  16. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  17. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  18. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  19. 2023Language-Driven Representation Learning for Roboticscited by 163
  20. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  21. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  22. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  23. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  24. 2023Structured World Models from Human Videoscited by 144
  25. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  26. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 135
  27. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  28. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  29. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  30. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  31. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  32. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  33. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  34. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  35. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  36. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  37. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  38. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106

50 most cited of 545 All 545 (JSON)