mlsanityData to 7 Oct 2026

RelationField: Relate Anything in Radiance Fields

arXiv 2412.13652 ↗First posted 18 Dec 2024cs.CVcited by 16

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Neural radiance fields are an emerging 3D scene representation and recently even been extended to learn features for scene understanding by distilling open-vocabulary features from vision-language models. However, current method primarily focus on object-centric representations, supporting object segmentation or detection, while understanding semantic relationships between objects remains largely unexplored. To address this gap, we propose RelationField, the first method to extract inter-object relationships directly from neural radiance fields. RelationField represents relationships between objects as pairs of rays within a neural radiance field, effectively extending its formulation to include implicit relationship queries. To teach RelationField complex, open-vocabulary relationships, relationship knowledge is distilled from multi-modal LLMs. To evaluate RelationField, we solve open-vocabulary 3D scene graph generation tasks and relationship-guided instance segmentation, achieving state-of-the-art performance in both tasks. See the project website at https://relationfield.github.io.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 65 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

55 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views4 Apr
  2. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models23 Jul

Other references · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Segment Anythingcited by 8,158
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  11. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  12. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
38 more
  1. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  3. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  4. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  5. 2023LERF: Language Embedded Radiance Fieldscited by 538
  6. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  7. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  8. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  9. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  10. 2016Visual Relationship Detection with Language Priorscited by 333
  11. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  12. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  13. 2018Graph R-CNN for Scene Graph Generationcited by 296
  14. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  15. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  16. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  17. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  18. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  19. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 175
  20. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 171
  21. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 159
  22. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  23. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
  24. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  25. 2024GARField: Group Anything with Radiance Fieldscited by 108
  26. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  27. 2024Clio: Real-time Task-Driven Open-Set 3D Scene Graphscited by 81
  28. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  29. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  30. 2024Click-Gaussian: Interactive Segmentation to Any 3D Gaussianscited by 58
  31. 2023NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detectioncited by 51
  32. 2022NeRF-RPN: A general framework for object detection in NeRFscited by 46
  33. 2023VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloudcited by 42
  34. 2023Incremental 3D Semantic Scene Graph Prediction from RGB Sequencescited by 39
  35. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  36. 2023SGAligner : 3D Scene Alignment with Scene Graphscited by 30
  37. 20193D-RelNet: Joint Object and Relational Network for 3D Predictioncited by 28
  38. 20223D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphscited by 24

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025Describe Anything Anywhere At Any Momentcited by 19
  2. 2026Multiplayer Interactive World Models with Representation Autoencoderscited by 14
  3. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 8
  4. 2025Unified Semantic Transformer for 3D Scene Understandingcited by 7
  5. 2026ReLaGS: Relational Language Gaussian Splattingcited by 6
  6. 2025OpenHype: Hyperbolic Embeddings for Hierarchical Open-Vocabulary Radiance Fieldscited by 5
  7. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  8. 2025Point2Act: Efficient 3D Distillation of Multimodal LLMs for Zero-Shot Context-Aware Graspingcited by 2
  9. 2025Remove360: Benchmarking Residuals After Object Removal in 3D Gaussian Splattingcited by 2
  10. 2025Is there anything left? Measuring semantic residuals of objects removed from 3D Gaussian Splattingcited by 1
  11. 2026EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understandingcited by 1
  12. 2026Analogical Trajectory Transfercited by 1
4 more
  1. 2026WildGHand: Learning Anti-Perturbation Gaussian Hand Avatars from Monocular In-the-Wild Videoscited by 0
  2. 2026NoPA: Non-Parametric Online 3D Scene Graph Generationcited by 0
  3. 2026GraphWrit3R: End-to-End 3D Scene Graph Writingcited by 0
  4. 2026RelationVGGT: Visual Geometry Transformers for 3D Spatial Relation Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis
  2. 2024RelationVLM: Making Large Vision-Language Models Understand Visual Relations
  3. 2024OpenObj: Open-Vocabulary Object-Level Neural Radiance Fields with Fine-Grained Understanding
  4. 2026Semantic Radiance Fields as Simulators for Spatial Reasoning in Real-World Scenes
  5. 2026RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs
  6. 2025Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generation
  7. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations
  8. 2025CoPa-SG: Dense Scene Graphs with Parametric and Proto-Relations