mlsanityData to 7 Oct 2026

Volumetric Environment Representation for Vision-Language Navigation

arXiv 2403.14158 ↗First posted 21 Mar 2024cs.CVcited by 63

Not on the map. Why?

Abstract (arXiv)

Vision-language navigation (VLN) requires an agent to navigate through an 3D environment based on visual observations and natural language instructions. It is clear that the pivotal factor for successful navigation lies in the comprehensive scene understanding. Previous VLN agents employ monocular frameworks to extract 2D features of perspective views directly. Though straightforward, they struggle for capturing 3D geometry and semantics, leading to a partial and incomplete environment representation. To achieve a comprehensive 3D representation with fine-grained details, we introduce a Volumetric Environment Representation (VER), which voxelizes the physical world into structured 3D cells. For each cell, VER aggregates multi-view 2D features into such a unified 3D space via 2D-3D sampling. Through coarse-to-fine feature extraction and multi-task learning for VER, our agent predicts 3D occupancy, 3D room layout, and 3D bounding boxes jointly. Based on online collected VERs, our agent performs volume state estimation and builds episodic memory for predicting the next step. Experimental results show our environment representations from multi-task learning lead to evident performance gains on VLN. Our model achieves state-of-the-art performance across VLN benchmarks (R2R, REVERIE, and R4R).

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 90 of its 109 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

90 of 109 references.

References · 90

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  10. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  11. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  12. 2019Objects as Pointscited by 1,172
38 more
  1. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  3. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  4. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  5. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  6. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  7. 2020Local Implicit Grid Representations for 3D Scenescited by 372
  8. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  9. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  10. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  11. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  12. 2020Learning to Explore using Active Neural SLAMcited by 308
  13. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  14. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  15. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  16. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  17. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  18. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  19. 2023Scene as Occupancycited by 208
  20. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  21. 2020Neural Topological SLAM for Visual Navigationcited by 193
  22. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  23. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  24. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  25. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  26. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  27. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  28. 2020Pillar-based Object Detection for Autonomous Drivingcited by 137
  29. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  30. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  31. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  32. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  33. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  34. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  35. 2023AerialVLN: Vision-and-Language Navigation for UAVscited by 117
  36. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  37. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  38. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 24; 2026: 24.

63 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  3. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 66
  4. 2024Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrievalcited by 36
  5. 2025OctoNav: Towards Generalist Embodied Navigationcited by 33
  6. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  7. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  8. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  9. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  10. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  11. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  12. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
38 more
  1. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  2. 2025Aerial Vision-and-Language Navigation with Grid-based View Selection and Map Constructioncited by 12
  3. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  4. 2024Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Spacecited by 10
  5. 2026Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Predictioncited by 9
  6. 2025Occupancy World Model for Robotscited by 8
  7. 2025Active Test-time Vision-Language Navigationcited by 7
  8. 2025VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulationcited by 7
  9. 2024MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillationcited by 6
  10. 2025PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigationcited by 6
  11. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  12. 2025Delta-Triplane Transformers as Occupancy World Modelscited by 5
  13. 2025CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigationcited by 5
  14. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5
  15. 2025History-Enhanced Two-Stage Transformer for Aerial Vision-and-Language Navigationcited by 5
  16. 2026Monocular Open Vocabulary Occupancy Prediction for Indoor Scenescited by 5
  17. 2026AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agentscited by 5
  18. 2024Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigationcited by 4
  19. 2025Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectioncited by 4
  20. 2025VPN: Visual Prompt Navigationcited by 4
  21. 2025OVSeg3R: Learn Open-vocabulary Instance Segmentation from 2D via 3D Reconstructioncited by 4
  22. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 4
  23. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4
  24. 2025Expand Your SCOPE: Semantic Cognition over Potential-Based Exploration for Embodied Visual Navigationcited by 3
  25. 2026Let's Reward Step-by-Step: Step-Aware Contrastive Alignment for Vision-Language Navigation in Continuous Environmentscited by 3
  26. 2026Stop Wandering: Efficient Vision-Language Navigation via Metacognitive Reasoningcited by 3
  27. 2026LookasideVLN: Direction-Aware Aerial Vision-and-Language Navigationcited by 3
  28. 2024World-Consistent Data Generation for Vision-and-Language Navigationcited by 2
  29. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  30. 2026WorldMAP: Bootstrapping Vision-Language Navigation Trajectory Prediction with Generative World Modelscited by 2
  31. 2026Turning Adaptation into Assets: Cross-Domain Bridging for Online Vision-Language Navigationcited by 2
  32. 2026Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Actioncited by 2
  33. 2025Fine-Grained Instruction-Guided Graph Reasoning for Vision-and-Language Navigationcited by 1
  34. 2025Continual Adaptation: Environment-Conditional Parameter Generation for Object Detection in Dynamic Scenarioscited by 1
  35. 2025Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understandingcited by 1
  36. 2025Decision-Driven Semantic Object Exploration for Legged Robots via Confidence-Calibrated Perception and Topological Subgoal Selectioncited by 1
  37. 2026SpatialFly: Implicit 3D Prior-Guided Visual Reparameterization for Continuous UAV Vision-and-Language Navigationcited by 1
  38. 2026Uncertainty-Aware Gaussian Map for Vision-Language Navigationcited by 1

50 most cited of 63 All 63 (JSON)