# Embodied Language Grounding with 3D Visual Feature Representations

arXiv: https://arxiv.org/abs/1910.01210 · HTML: https://mlsanity.com/paper/1910.01210/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2019.

## Built on by (most cited first, 20 of 20)

- [FILM: Following Instructions in Language with Modular Methods](https://mlsanity.com/paper/2110.07342/) (2021)
- [EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding](https://mlsanity.com/paper/2209.14941/) (2022)
- [Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding](https://mlsanity.com/paper/2309.04561/) (2023)
- [Fast and Explicit Neural View Synthesis](https://mlsanity.com/paper/2107.05775/) (2021)
- [Voxel-informed Language Grounding](https://mlsanity.com/paper/2205.09710/) (2022)
- [Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following](https://mlsanity.com/paper/2304.03767/) (2023)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Description](https://mlsanity.com/paper/2412.01398/) (2024)
- [LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following](https://mlsanity.com/paper/2310.12344/) (2023)
- [OpenBox: Annotate Any Bounding Boxes in 3D](https://mlsanity.com/paper/2512.01352/) (2025)
- [DoRO: Disambiguation of referred object for embodied agents](https://mlsanity.com/paper/2207.14205/) (2022)
- [Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data](https://mlsanity.com/paper/2411.15657/) (2024)
- [gComm: An environment for investigating generalization in Grounded Language Acquisition](https://mlsanity.com/paper/2105.03943/) (2021)
- [Bayesian Self-Training for Semi-Supervised 3D Segmentation](https://mlsanity.com/paper/2409.08102/) (2024)
- [Fine-Grained Spatial and Verbal Losses for 3D Visual Grounding](https://mlsanity.com/paper/2411.03405/) (2024)
- [LidaRefer: Context-aware Outdoor 3D Visual Grounding for Autonomous Driving](https://mlsanity.com/paper/2411.04351/) (2024)
- [ESCAPE: Episodic Spatial Memory and Adaptive Execution Policy for Long-Horizon Mobile Manipulation](https://mlsanity.com/paper/2604.13633/) (2026)
- [Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language](https://mlsanity.com/paper/2511.15308/) (2025)
- [Auto-Vocabulary 3D Object Detection](https://mlsanity.com/paper/2512.16077/) (2025)
- [S4VY: Segment Anything in Feed-Forward 4D Visual Geometry](https://mlsanity.com/paper/2609.36875/) (2026)

## Abstract (arXiv)

We propose associating language utterances to 3D visual abstractions of the scene they describe. The 3D visual abstractions are encoded as 3-dimensional visual feature maps. We infer these 3D visual scene feature maps from RGB images of the scene via view prediction: when the generated 3D scene feature map is neurally projected from a camera viewpoint, it should match the corresponding RGB image. We present generative models that condition on the dependency tree of an utterance and generate a corresponding visual 3D feature map as well as reason about its plausibility, and detector models that condition on both the dependency tree of an utterance and a related image and localize the object referents in the 3D feature map inferred from the image. Our model outperforms models of language and vision that associate language with 2D CNN activations or 2D images by a large margin in a variety of tasks, such as, classifying plausibility of utterances, detecting referential expressions, and supplying rewards for trajectory optimization of object placement policies from language instructions. We perform numerous ablations and show the improved performance of our detectors is due to its better generalization across camera viewpoints and lack of object interferences in the inferred 3D feature space, and the improved performance of our generators is due to their ability to spatially reason about objects and their configurations in 3D when mapping from language to scenes.