mlsanityData to 7 Oct 2026

Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping

arXiv 2309.07970 ↗First posted 14 Sep 2023cs.RO cs.CVcited by 91

Not on the map. Why?

Abstract (arXiv)

Grasping objects by a specific part is often crucial for safety and for executing downstream tasks. Yet, learning-based grasp planners lack this behavior unless they are trained on specific object part data, making it a significant challenge to scale object diversity. Instead, we propose LERF-TOGO, Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which uses vision-language models zero-shot to output a grasp distribution over an object given a natural language query. To accomplish this, we first reconstruct a LERF of the scene, which distills CLIP embeddings into a multi-scale 3D language field queryable with text. However, LERF has no sense of objectness, meaning its relevancy outputs often return incomplete activations over an object which are insufficient for subsequent part queries. LERF-TOGO mitigates this lack of spatial grouping by extracting a 3D object mask via DINO features and then conditionally querying LERF on this mask to obtain a semantic distribution over the object with which to rank grasps from an off-the-shelf grasp planner. We evaluate LERF-TOGO's ability to grasp task-oriented object parts on 31 different physical objects, and find it selects grasps on the correct part in 81% of all trials and grasps successfully in 69%. See the project website at: lerftogo.github.io

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 70 of its 91 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

70 of 91 references.

References · 70

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  5. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  8. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  9. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  10. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  11. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  12. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
38 more
  1. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  3. 2022Language-driven Semantic Segmentationcited by 711
  4. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  5. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  6. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  7. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  8. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  9. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  10. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  11. 2021NICE-SLAM: Neural Implicit Scalable Encoding for SLAMcited by 586
  12. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  13. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  14. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  15. 2023LERF: Language Embedded Radiance Fieldscited by 538
  16. 2021iMAP: Implicit Mapping and Positioning in Real-Timecited by 476
  17. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  18. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  19. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  20. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  21. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  22. 2022Visual Language Maps for Robot Navigationcited by 362
  23. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  24. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  25. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  26. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  27. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  28. 2023SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesiscited by 247
  29. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  30. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  31. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  32. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  33. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  34. 2022NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fieldscited by 188
  35. 2021Deblur-NeRF: Neural Radiance Fields from Blurry Imagescited by 179
  36. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  37. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  38. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 31; 2025: 34; 2026: 18.

91 papers.

Sort:

By area:

  1. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  2. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  3. 2023On Bringing Robots Homecited by 101
  4. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  5. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
  6. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  7. 2023Language Models as Zero-Shot Trajectory Generatorscited by 59
  8. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  9. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 51
  10. 2025SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretrainingcited by 51
  11. 2025Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registrationcited by 50
  12. 2023One-Shot Open Affordance Learning with Foundation Modelscited by 45
38 more
  1. 2024Physical Property Understanding from Language-Embedded Feature Fieldscited by 45
  2. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  3. 2024SelfSplat: Pose-Free and 3D Prior-Free Generalizable 3D Gaussian Splattingcited by 40
  4. 2024Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splattingcited by 37
  5. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  6. 2024Learning Precise Affordances from Egocentric Videos for Robotic Manipulationcited by 34
  7. 2024DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Featurescited by 33
  8. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 32
  9. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  10. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  11. 2023SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulationcited by 22
  12. 2025GaussGym: An open-source real-to-sim framework for learning locomotion from pixelscited by 22
  13. 2024Neural Fields in Robotics: A Surveycited by 21
  14. 2025Viser: Imperative, Web-based 3D Visualization in Pythoncited by 19
  15. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  16. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  17. 2024Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidancecited by 16
  18. 2025Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learningcited by 16
  19. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  20. 2024ShapeGrasp: Zero-Shot Task-Oriented Grasping with Large Language Models through Geometric Decompositioncited by 14
  21. 2025GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulationcited by 14
  22. 2025SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splattingcited by 14
  23. 2024Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Spacecited by 12
  24. 2024INTRA: Interaction Relationship-aware Weakly Supervised Affordance Groundingcited by 12
  25. 2024Multi-GraspLLM: A Multimodal LLM for Multi-Hand Semantic Guided Grasp Generationcited by 12
  26. 2025RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Graspingcited by 11
  27. 2024MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenescited by 9
  28. 2024Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMscited by 8
  29. 2025COS3D: Collaborative Open-Vocabulary 3D Segmentationcited by 8
  30. 2024HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulationcited by 7
  31. 2024HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Modelscited by 7
  32. 2024FAST-Splat: Fast, Ambiguity-Free Semantics Transfer in Gaussian Splattingcited by 7
  33. 2025Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splattingcited by 7
  34. 2025LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusioncited by 6
  35. 2024Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Graspingcited by 5
  36. 2024SwarmGPT: Combining Large Language Models with Safe Motion Planning for Drone Swarm Choreographycited by 5
  37. 2025SIREN: Semantic, Initialization-Free Registration of Multi-Robot Gaussian Splatting Mapscited by 5
  38. 2025GrowSplat: Constructing Temporal Digital Twins of Plants with Gaussian Splatscited by 5

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Open-World Grasping with Large Vision-Language Models
  2. 2026SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments
  3. 2025MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping
  4. 2025Attribute-based Object Grounding and Robot Grasp Detection with Spatial Reasoning
  5. 2024A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping
  6. 2024GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Grasping
  7. 2026ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning
  8. 2025VLAD-Grasp: Zero-shot Grasp Detection via Vision-Language Models