mlsanityData to 7 Oct 2026

Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation

arXiv 2306.17817 ↗First posted 30 Jun 2023cs.RO cs.AI cs.LGcited by 135

Not on the map. Why?

Cited by 30 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

3D perceptual representations are well suited for robot manipulation as they easily encode occlusions and simplify spatial reasoning. Many manipulation tasks require high spatial precision in end-effector pose prediction, which typically demands high-resolution 3D feature grids that are computationally expensive to process. As a result, most manipulation policies operate directly in 2D, foregoing 3D inductive biases. In this paper, we introduce Act3D, a manipulation policy transformer that represents the robot's workspace using a 3D feature field with adaptive resolutions dependent on the task at hand. The model lifts 2D pre-trained features to 3D using sensed depth, and attends to them to compute features for sampled 3D points. It samples 3D point grids in a coarse to fine manner, featurizes them using relative-position attention, and selects where to focus the next round of point sampling. In this way, it efficiently computes 3D action maps of high spatial resolution. Act3D sets a new state-of-the-art in RL-Bench, an established manipulation benchmark, where it achieves 10% absolute improvement over the previous SOTA 2D multi-view policy on 74 RLBench tasks and 22% absolute improvement with 3x less compute over the previous SOTA 3D policy. We quantify the importance of relative spatial attention, large-scale vision-language pre-trained 2D backbones, and weight tying across coarse-to-fine attentions in ablative experiments. Code and videos are available on our project website: https://act3d.github.io/.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

30 of 37 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  6. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2022A Generalist Agentcited by 802
  10. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
18 more
  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  2. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  3. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  4. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  5. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  6. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 111
  7. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  8. 2014Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Studycited by 86
  9. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  10. 2020Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networkscited by 77
  11. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  12. 2015Sparse 3D convolutional neural networkscited by 71
  13. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 50
  14. 2019Graph-Structured Visual Imitationcited by 43
  15. 2018Learning Spatial Common Sense with Geometry-Aware Recurrent Networkscited by 35
  16. 2023Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangementcited by 30
  17. 20203D-OES: Viewpoint-Invariant Object-Factorized Environment Simulatorscited by 21
  18. 2021HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworkscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 34; 2025: 46; 2026: 51.

135 papers.

Sort:

By area:

  1. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  2. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  4. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  5. 2025History-Guided Video Diffusioncited by 165
  6. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  7. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  8. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  9. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  10. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  11. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  12. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
38 more
  1. 2024Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matchingcited by 63
  2. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  3. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modificationcited by 49
  4. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 46
  5. 2024Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learningcited by 45
  6. 2024VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulationcited by 45
  7. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  8. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  9. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  10. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  11. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  12. 2024G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulationcited by 35
  13. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 33
  14. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
  15. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  16. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  17. 2025GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Modelcited by 24
  18. 2024VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulationcited by 21
  19. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulationcited by 21
  20. 2026HoloBrain-0 Technical Reportcited by 20
  21. 2024Render and Diffuse: Aligning Image and Action Spaces for Diffusion-based Behaviour Cloningcited by 19
  22. 20253D FlowMatch Actor: Unified 3D Policy for Single- and Dual-Arm Manipulationcited by 19
  23. 2025QDepth-VLA: Quantized Depth Prediction as Auxiliary Supervision for Vision-Language-Action Modelscited by 19
  24. 2024One-Shot Imitation Learning with Invariance Matching for Robotic Manipulationcited by 18
  25. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  26. 2024Keypoint Abstraction using Large Models for Object-Relative Imitation Learningcited by 18
  27. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulationcited by 18
  28. 2024PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulationcited by 17
  29. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 17
  30. 2025Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learningcited by 16
  31. 2025Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioningcited by 14
  32. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 14
  33. 2026ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulationcited by 14
  34. 2024Leveraging Locality to Boost Sample Efficiency in Robotic Manipulationcited by 13
  35. 2025FlowRAM: Grounding Flow Matching Policy with Region-Aware Mamba Framework for Robotic Manipulationcited by 13
  36. 2026Any3D-VLA: Enhancing VLA Robustness via Diverse Point Cloudscited by 13
  37. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  38. 2025GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulationcited by 11

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Robo3R: Enhancing Robotic Manipulation with Accurate Feed-Forward 3D Reconstruction
  2. 2026Bridge3D: Enabling Vision-Language-Action Models to See and Act in 3D
  3. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation
  4. 2026Dexterity-BEV: Aligning 3D World and Actions for Generalizable Robot Policies Learning
  5. 2026Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation
  6. 2026VolumeDP: Modeling Volumetric Representation for Manipulation Policy Learning
  7. 2026Robotic Manipulation is Vision-to-Geometry Mapping: Vision-Geometry Backbones over Language and Video Models
  8. 2025Seeing the Bigger Picture: 3D Latent Mapping for Mobile Manipulation Policy Learning