mlsanityData to 7 Oct 2026

Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Vision

arXiv 2207.01136 ↗First posted 3 Jul 2022cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper focuses on perceiving and navigating 3D environments using echoes and RGB image. In particular, we perform depth estimation by fusing RGB image with echoes, received from multiple orientations. Unlike previous works, we go beyond the field of view of the RGB and estimate dense depth maps for substantially larger parts of the environment. We show that the echoes provide holistic and in-expensive information about the 3D structures complementing the RGB image. Moreover, we study how echoes and the wide field-of-view depth maps can be utilised in robot navigation. We compare the proposed methods against recent baselines using two sets of challenging realistic 3D environments: Replica and Matterport3D. The implementation and pre-trained models will be made publicly available.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 78 of its 105 references that are in the map. First posted on arXiv 3 Jul 2022.

Builds on

78 of 105 references.

References · 78

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  5. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  9. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  10. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  11. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  12. 2018On Evaluation of Embodied Navigation Agentscited by 526
38 more
  1. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  2. 2017Look, Listen and Learncited by 452
  3. 2019From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimationcited by 379
  4. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  5. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  6. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  7. 2020Learning to Explore using Active Neural SLAMcited by 308
  8. 2018The Sound of Pixelscited by 258
  9. 2017Objects that Soundcited by 254
  10. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  11. 2019Vision-and-Dialog Navigationcited by 246
  12. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  13. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  14. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  15. 2018Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentationcited by 222
  16. 2020Consistent Video Depth Estimationcited by 220
  17. 2020Neural Topological SLAM for Visual Navigationcited by 193
  18. 2016Learning to Navigate in Complex Environmentscited by 192
  19. 2018Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraintscited by 179
  20. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  21. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  22. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  23. 2019The Sound of Motionscited by 145
  24. 2018Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundariescited by 144
  25. 2019Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Cameracited by 130
  26. 2021Transformer-Based Attention Networks for Continuous Pixel-Wise Predictioncited by 125
  27. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  28. 2019Learning Exploration Policies for Navigationcited by 113
  29. 2014Deep Convolutional Neural Fields for Depth Estimation from a Single Imagecited by 106
  30. 2021Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Mergingcited by 104
  31. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  32. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  33. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  34. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  35. 2020Guiding Monocular Depth Estimation Using Depth-Attention Volumecited by 82
  36. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  37. 2020Semantic Audio-Visual Navigationcited by 79
  38. 2014Material Recognition in the Wild with the Materials in Context Databasecited by 77

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fieldscited by 8
  2. 2026Few-shot Acoustic Synthesis with Multimodal Flow Matchingcited by 2
  3. 2024Estimating Indoor Scene Depth Maps from Ultrasonic Echoescited by 0
  4. 2024AVS-Net: Audio-Visual Scale Net for Self-supervised Monocular Metric Depth Estimationcited by 0