mlsanityData to 7 Oct 2026

Learning Generalizable Feature Fields for Mobile Manipulation

arXiv 2403.07563 ↗First posted 12 Mar 2024cs.RO cs.CV cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

An open problem in mobile manipulation is how to represent objects and scenes in a unified manner so that robots can use both for navigation and manipulation. The latter requires capturing intricate geometry while understanding fine-grained semantics, whereas the former involves capturing the complexity inherent at an expansive physical scale. In this work, we present GeFF (Generalizable Feature Fields), a scene-level generalizable neural feature field that acts as a unified representation for both navigation and manipulation that performs in real-time. To do so, we treat generative novel view synthesis as a pre-training task, and then align the resulting rich scene priors with natural language via CLIP feature distillation. We demonstrate the effectiveness of this approach by deploying GeFF on a quadrupedal robot equipped with a manipulator. We quantitatively evaluate GeFF's ability for open-vocabulary object-/part-level manipulation and show that GeFF outperforms point-based baselines in runtime and storage-accuracy trade-offs, with qualitative examples of semantics-aware navigation and articulated object manipulation.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 70 of its 84 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

70 of 84 references.

References · 70

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023Segment Anythingcited by 8,158
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  11. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  12. 2020Point Transformercited by 1,462
38 more
  1. 2020Implicit Geometric Regularization for Learning Shapescited by 707
  2. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  3. 2021Extract Free Dense Labels from CLIPcited by 580
  4. 2023LERF: Language Embedded Radiance Fieldscited by 538
  5. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  6. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  7. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  8. 2021Advances in Neural Renderingcited by 365
  9. 2022Visual Language Maps for Robot Navigationcited by 362
  10. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  11. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  12. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  13. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  14. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  15. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  16. 20203D Object Detection with Pointformercited by 194
  17. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  18. 2020GRF: Learning a General Radiance Field for 3D Representation and Renderingcited by 183
  19. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  20. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  21. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  22. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  23. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  24. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  25. 2023F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectoriescited by 118
  26. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 117
  27. 2022Is Attention All That NeRF Needs?cited by 113
  28. 2022iSDF: Real-Time Neural Signed Distance Fields for Robot Perceptioncited by 107
  29. 2022Navigating to Objects in the Real Worldcited by 104
  30. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  31. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  32. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  33. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  34. 2021LOLNeRF: Learn from One Lookcited by 84
  35. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  36. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  37. 2021Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulationcited by 59
  38. 2021Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systemscited by 53

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 17; 2026: 1.

31 papers.

Sort:

By area:

  1. 2025Humanoid Policy ~ Human Policycited by 65
  2. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  3. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  4. 2025MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulationcited by 34
  5. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  6. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  7. 2024Neural Fields in Robotics: A Surveycited by 21
  8. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  9. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  10. 2024GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Graspingcited by 15
  11. 2025GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulationcited by 14
  12. 2024One Map to Find Them All: Real-time Open-Vocabulary Mapping for Zero-shot Multi-Object Navigationcited by 13
19 more
  1. 2025SpatialSplat: Efficient Semantic 3D from Sparse Unposed Imagescited by 13
  2. 2025FreeSplat++: Generalizable 3D Gaussian Splatting for Efficient Indoor Scene Reconstructioncited by 12
  3. 2025ODYSSEY: Open-World Quadrupeds Exploration and Manipulation for Long-Horizon Taskscited by 9
  4. 2025Human2LocoMan: Learning Versatile Quadrupedal Manipulation with Human Pretrainingcited by 8
  5. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguitycited by 8
  6. 2025MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planningcited by 8
  7. 2024OpenObj: Open-Vocabulary Object-Level Neural Radiance Fields with Fine-Grained Understandingcited by 6
  8. 2024WildLMa: Long Horizon Loco-Manipulation in the Wildcited by 6
  9. 2025SIREN: Semantic, Initialization-Free Registration of Multi-Robot Gaussian Splatting Mapscited by 5
  10. 2025DETACH: Cross-domain Learning for Long-Horizon Tasks via Mixture of Disentangled Expertscited by 5
  11. 2024Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigationcited by 4
  12. 2025SLIM: Sim-to-Real Legged Instructive Manipulation via Long-Horizon Visuomotor Learningcited by 4
  13. 2025OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesiscited by 3
  14. 2025Prof. Robot: Differentiable Robot Rendering Without Static and Self-Collisionscited by 2
  15. 2024From Words to Poses: Enhancing Novel Object Pose Estimation with Vision Language Modelscited by 1
  16. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  17. 2024EUGens: Efficient, Unified, and General Dense Layerscited by 0
  18. 2025UniFField: A Generalizable Unified Neural Feature Field for Visual, Semantic, and Spatial Uncertainties in Any Scenecited by 0
  19. 2026EUGens: Efficient, Unified, and General Dense Layerscited by 0