mlsanityData to 14 Oct 2026

Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation

arXiv 2308.07931 ↗First posted 27 Jul 2023cs.CV cs.AI cs.CLcited by 130

Not on the map. Why?

Abstract (arXiv)

Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manipulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models. We present a few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using features distilled from a vision-language model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize to unseen expressions and novel categories of objects.

Key facts. Cited by 130 papers in the map (data through 14 Oct 2026). Builds on 49 of its 69 references that are in the map. First posted on arXiv 27 Jul 2023.

Builds on

49 of 69 references.

References · 49

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  10. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  11. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 642
  12. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
37 more
  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  2. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 600
  3. 2021Extract Free Dense Labels from CLIPcited by 583
  4. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  5. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  6. 2023LERF: Language Embedded Radiance Fieldscited by 543
  7. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  8. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  9. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  10. 2022Visual Language Maps for Robot Navigationcited by 365
  11. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  12. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  13. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 264
  14. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  15. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
  16. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  17. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  18. 2023Language-Driven Representation Learning for Roboticscited by 165
  19. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 156
  20. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 142
  21. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  22. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusioncited by 127
  23. 2022Is Attention All That NeRF Needs?cited by 113
  24. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 107
  25. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  26. 2016Universal Correspondence Networkcited by 82
  27. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  28. 2022Compressing Volumetric Radiance Fields to 1 MBcited by 68
  29. 2022GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRFcited by 67
  30. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  31. 2023Neural Volumetric Memory for Visual Locomotion Controlcited by 44
  32. 2022Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understandingcited by 37
  33. 2018Learning Spatial Common Sense with Geometry-Aware Recurrent Networkscited by 35
  34. 2019RTFM: Generalising to Novel Environment Dynamics via Readingcited by 33
  35. 2022MIRA: Mental Imagery for Robotic Affordancescited by 26
  36. 20203D-OES: Viewpoint-Invariant Object-Factorized Environment Simulatorscited by 21
  37. 2023USA-Net: Unified Semantic and Affordance Representations for Robot Memorycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 50; 2025: 49; 2026: 25.

130 papers.

Sort:

By area:

  1. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  2. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 364
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  4. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  5. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  6. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 116
  7. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  8. 2023On Bringing Robots Homecited by 101
  9. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 78
  10. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  11. 2024Large Spatial Model: End-to-end Unposed Images to Semantic 3Dcited by 73
  12. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 71
38 more
  1. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 65
  2. 2023Language Models as Zero-Shot Trajectory Generatorscited by 60
  3. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 57
  4. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 54
  5. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  6. 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 47
  7. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  8. 2025LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPScited by 41
  9. 2024Denoising Vision Transformerscited by 40
  10. 2024Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splattingcited by 38
  11. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  12. 2024G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulationcited by 35
  13. 20254D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Modelscited by 35
  14. 2024Learning Precise Affordances from Egocentric Videos for Robotic Manipulationcited by 34
  15. 2024Find Any Part in 3Dcited by 32
  16. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  17. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  18. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 27
  19. 2024ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matchingcited by 26
  20. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  21. 2025Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixelscited by 24
  22. 2025Lifting by Gaussians: A Simple, Fast and Flexible Method for 3D Instance Segmentationcited by 23
  23. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 22
  24. 2025GaussGym: An open-source real-to-sim framework for learning locomotion from pixelscited by 22
  25. 2024Neural Fields in Robotics: A Surveycited by 21
  26. 2025C3G: Learning Compact 3D Representations with 2K Gaussianscited by 21
  27. 2024NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fieldscited by 20
  28. 2024Keypoint Abstraction using Large Models for Object-Relative Imitation Learningcited by 20
  29. 2024When Does Perceptual Alignment Benefit Vision Representations?cited by 18
  30. 2024GAGS: Granularity-Aware Feature Distillation for Language Gaussian Splattingcited by 18
  31. 2024PLGS: Robust Panoptic Lifting with 3D Gaussian Splattingcited by 17
  32. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  33. 2024NeRFs in Robotics: A Surveycited by 16
  34. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  35. 2024GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Graspingcited by 15
  36. 2024GSemSplat: Generalizable Semantic 3D Gaussian Splatting from Uncalibrated Image Pairscited by 15
  37. 2025GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulationcited by 14
  38. 2025SemanticSplat: Feed-Forward 3D Scene Understanding with Language-Aware Gaussian Fieldscited by 14

50 most cited of 130 All 130 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation
  2. 2025Improving Generalization of Language-Conditioned Robot Manipulation
  3. 2025MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping
  4. 2025Free-form language-based robotic reasoning and grasping
  5. 2026Robotic Manipulation is Vision-to-Geometry Mapping: Vision-Geometry Backbones over Language and Video Models
  6. 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
  7. 2026Learning Part-Aware Dense 3D Feature Field for Generalizable Articulated Object Manipulation
  8. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation