mlsanityData to 14 Oct 2026

Bird's-Eye-View Scene Graph for Vision-Language Navigation

arXiv 2308.04758 ↗First posted 9 Aug 2023cs.CV cs.AIcited by 72

Not on the map. Why?

Abstract (arXiv)

Vision-language navigation (VLN), which entails an agent to navigate 3D environments following human instructions, has shown great advances. However, current agents are built upon panoramic observations, which hinders their ability to perceive 3D scene geometry and easily leads to ambiguous selection of panoramic view. To address these limitations, we present a BEV Scene Graph (BSG), which leverages multi-step BEV representations to encode scene layouts and geometric cues of indoor environment under the supervision of 3D detection. During navigation, BSG builds a local BEV representation at each step and maintains a BEV-based global scene map, which stores and organizes all the online collected local BEV representations according to their topological relations. Based on BSG, the agent predicts a local BEV grid-level decision score and a global graph-level decision score, combined with a sub-view selection score on panoramic views, for more accurate action prediction. Our approach significantly outperforms state-of-the-art methods on REVERIE, R2R, and R4R, showing the potential of BEV perception in VLN.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Builds on 113 of its 143 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

113 of 143 references.

References · 113

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
38 more
  1. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  5. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  8. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  10. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  11. 2017Focal Loss for Dense Object Detectioncited by 752
  12. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
  13. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  14. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 658
  15. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  16. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  17. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 572
  18. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  19. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  20. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  21. 2016Semantic Scene Completion from a Single Depth Imagecited by 472
  22. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  23. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  24. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  25. 2020Local Implicit Grid Representations for 3D Scenescited by 373
  26. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  27. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 341
  28. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  29. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 336
  30. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  31. 2021Categorical Depth Distribution Network for Monocular 3D Object Detectioncited by 327
  32. 2020Learning to Explore using Active Neural SLAMcited by 316
  33. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  34. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  35. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  36. 2021Is Pseudo-Lidar needed for Monocular 3D Object detection?cited by 249
  37. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  38. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231

50 most cited of 113 All 113 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 24; 2025: 29; 2026: 16.

72 papers.

Sort:

By area:

  1. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 98
  2. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 85
  3. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  4. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 65
  5. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  6. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  7. 2024IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detectioncited by 57
  8. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 57
  9. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  10. 2024GaussNav: Gaussian Splatting for Visual Navigationcited by 49
  11. 2025OctoNav: Towards Generalist Embodied Navigationcited by 34
  12. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
38 more
  1. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  2. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 23
  3. 2024Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activitycited by 22
  4. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 22
  5. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 21
  6. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 21
  7. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 20
  8. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  9. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 19
  10. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  11. 2025MonoDream: Monocular Vision-Language Navigation with Panoramic Dreamingcited by 16
  12. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 15
  13. 2025DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigationcited by 14
  14. 2024Loc4Plan: Locating Before Planning for Outdoor Vision and Language Navigationcited by 11
  15. 2025DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generationcited by 11
  16. 2024Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generationcited by 10
  17. 2024Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Spacecited by 10
  18. 2025GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigationcited by 10
  19. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigationcited by 10
  20. 2024Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomalycited by 9
  21. 2024Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigationcited by 8
  22. 2024MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillationcited by 6
  23. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  24. 2025History-Enhanced Two-Stage Transformer for Aerial Vision-and-Language Navigationcited by 6
  25. 2025CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigationcited by 5
  26. 2025Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotationscited by 5
  27. 2025ObjectReact: Learning Object-Relative Control for Visual Navigationcited by 5
  28. 2026textscNaVIDA: Vision-Language Navigation with Inverse Dynamics Augmentationcited by 5
  29. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 5
  30. 2025Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectioncited by 4
  31. 2025Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigationcited by 4
  32. 2025Boosting Zero-Shot VLN via Abstract Obstacle Map-Based Waypoint Prediction with TopoGraph-and-VisitInfo-Aware Promptingcited by 4
  33. 2026MapDream: Task-Driven Map Learning for Vision-Language Navigationcited by 4
  34. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4
  35. 2024BEVNav: Robot Autonomous Navigation Via Spatial-Temporal Contrastive Learning in Bird's-Eye Viewcited by 3
  36. 2025FishBEV: Distortion-Resilient Bird's Eye View Segmentation with Surround-View Fisheye Camerascited by 3
  37. 2025Context and Diversity Matter: The Emergence of In-Context Learning in World Modelscited by 3
  38. 2025Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigationcited by 3

50 most cited of 72 All 72 (JSON)