mlsanityData to 14 Oct 2026

GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields

arXiv 2308.16891 ↗First posted 31 Aug 2023cs.RO cs.CV cs.LGcited by 99

Not on the map. Why?

Cited by 24 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

It is a long-standing problem in robotics to develop agents capable of executing diverse manipulation tasks from visual observations in unstructured real-world environments. To achieve this goal, the robot needs to have a comprehensive understanding of the 3D structure and semantics of the scene. In this work, we present GNFactor, a visual behavior cloning agent for multi-task robotic manipulation with Generalizable Neural feature Fields. GNFactor jointly optimizes a generalizable neural field (GNF) as a reconstruction module and a Perceiver Transformer as a decision-making module, leveraging a shared deep 3D voxel representation. To incorporate semantics in 3D, the reconstruction module utilizes a vision-language foundation model (textite.g., Stable Diffusion) to distill rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3 real robot tasks and perform detailed ablations on 10 RLBench tasks with a limited number of demonstrations. We observe a substantial improvement of GNFactor over current state-of-the-art methods in seen and unseen tasks, demonstrating the strong generalization ability of GNFactor. Our project website is https://yanjieze.com/GNFactor/ .

Key facts. Cited by 99 papers in the map (data through 14 Oct 2026). Builds on 53 of its 57 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

53 of 57 references.

References · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  7. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  8. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  11. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
38 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  2. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 703
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  4. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  6. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  7. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  8. 2020Learning Continuous Image Representation with Local Implicit Image Functioncited by 552
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  10. 2023LERF: Language Embedded Radiance Fieldscited by 543
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  12. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  13. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  14. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  15. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 242
  16. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  17. 2020GRF: Learning a General Radiance Field for 3D Representation and Renderingcited by 183
  18. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  19. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  20. 2021Efficient Neural Radiance Fields for Interactive Free-viewpoint Videocited by 150
  21. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
  22. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  23. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 122
  24. 20213D Neural Scene Representations for Visuomotor Controlcited by 111
  25. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 108
  26. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 107
  27. 2022Vision Transformer for NeRF-Based View Synthesis from a Single Input Imagecited by 104
  28. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  29. 2021ShaRF: Shape-conditioned Radiance Fields from a Single Viewcited by 92
  30. 2022Transformers as Meta-Learners for Implicit Neural Representationscited by 79
  31. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  32. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 66
  33. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 65
  34. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 56
  35. 2023FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Modelscited by 47
  36. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  37. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  38. 2022MIRA: Mental Imagery for Robotic Affordancescited by 26

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 42; 2025: 34; 2026: 17.

99 papers.

Sort:

By area:

  1. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 282
  2. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 133
  3. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 116
  4. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 108
  5. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 105
  6. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  7. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  8. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 78
  9. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 73
  10. 2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
  11. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 57
  12. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 52
38 more
  1. 2025ManiFlow: A General Robot Manipulation Policy via Consistency Flow Trainingcited by 49
  2. 2024Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learningcited by 45
  3. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  4. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 41
  5. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  6. 2024EmbodiedSAM: Online Segment Any 3D Thing in Real Timecited by 40
  7. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 39
  8. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  9. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  10. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  11. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  12. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 35
  13. 2025MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulationcited by 34
  14. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 32
  15. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  16. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  17. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  18. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 27
  19. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  20. 2024DNAct: Diffusion Guided Multi-Task 3D Policy Learningcited by 23
  21. 2025MoE-Loco: Mixture of Experts for Multitask Locomotioncited by 23
  22. 2025WristWorld: Generating Wrist-Views via 4D World Models for Robotic Manipulationcited by 23
  23. 2023SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulationcited by 22
  24. 2024MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoderscited by 22
  25. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 22
  26. 2024Neural Fields in Robotics: A Surveycited by 21
  27. 2024NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fieldscited by 20
  28. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 19
  29. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 19
  30. 2024Generative Image as Action Modelscited by 18
  31. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 17
  32. 2024Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidancecited by 16
  33. 2024AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulationcited by 16
  34. 2024Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulationcited by 15
  35. 2025FDPP: Fine-tune Diffusion Policy with Human Preferencecited by 15
  36. 2025Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioningcited by 14
  37. 2026Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulationcited by 14
  38. 2024Reinforcement Learning with Generalizable Gaussian Splattingcited by 13

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Transferring Foundation Models for Generalizable Robotic Manipulation
  2. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulation
  3. 2025Compose by Focus: Scene Graph-based Atomic Skills
  4. 2026TriRelVLA: Triadic Relational Structure for Generalizable Embodied Manipulation
  5. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation
  6. 2022VIMA: General Robot Manipulation with Multimodal Prompts
  7. 2025DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulation
  8. 2024Learning Generalizable 3D Manipulation With 10 Demonstrations