mlsanityData to 14 Oct 2026

Semantic Audio-Visual Navigation

arXiv 2012.11583 ↗First posted 21 Dec 2020cs.CV cs.LG cs.ROcited by 82

Not on the map. Why?

Abstract (arXiv)

Recent work on audio-visual navigation assumes a constantly-sounding target and restricts the role of audio to signaling the target's position. We introduce semantic audio-visual navigation, where objects in the environment make sounds consistent with their semantic meaning (e.g., toilet flushing, door creaking) and acoustic events are sporadic or short in duration. We propose a transformer-based model to tackle this new semantic AudioGoal task, incorporating an inferred goal descriptor that captures both spatial and semantic properties of the target. Our model's persistent multimodal memory enables it to reach the goal even long after the acoustic event stops. In support of the new task, we also expand the SoundSpaces audio simulations to provide semantically grounded sounds for an array of objects in Matterport3D. Our method strongly outperforms existing audio-visual navigation methods by learning to associate semantic, acoustic, and visual cues.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 23; 2023: 15; 2024: 5; 2025: 11; 2026: 19.

82 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 400
  2. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  3. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 150
  4. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 95
  5. 2022Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigationcited by 67
  6. 2021Learning to Map for Active Semantic Goal Navigationcited by 50
  7. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  8. 2022Sound Adversarial Audio-Visual Navigationcited by 48
  9. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  10. 2022Retrospectives on the Embodied AI Workshopcited by 46
  11. 2022Visual Acoustic Matchingcited by 45
  12. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
38 more
  1. 2021Toward Practical Monocular Indoor Depth Estimationcited by 42
  2. 2021Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Soundscited by 41
  3. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  4. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  5. 2023Audio-Visual Class-Incremental Learningcited by 33
  6. 2021Learning Audio-Visual Dereverberationcited by 26
  7. 2022AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environmentscited by 26
  8. 2022Towards Versatile Embodied Navigationcited by 26
  9. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  10. 2023Measuring Acoustics with Collaborative Multiple Agentscited by 22
  11. 2022Learning Active Camera for Multi-Object Navigationcited by 21
  12. 2025Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoningcited by 20
  13. 2023Omnidirectional Information Gathering for Knowledge Transfer-based Audio-Visual Navigationcited by 19
  14. 2022Pay Self-Attention to Audio-Visual Navigationcited by 18
  15. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  16. 2023Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parsercited by 18
  17. 2023Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigationcited by 18
  18. 2023XVO: Generalized Visual Odometry via Cross-Modal Self-Trainingcited by 18
  19. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17
  20. 2022Active Audio-Visual Separation of Dynamic Sound Sourcescited by 16
  21. 2023CASP-Net: Rethinking Video Saliency Prediction from an Audio-VisualConsistency Perceptual Perspectivecited by 15
  22. 2023AdVerb: Visually Guided Audio Dereverberationcited by 15
  23. 2025Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigationcited by 15
  24. 2022Finding Fallen Objects Via Asynchronous Audio-Visual Integrationcited by 14
  25. 2023Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoningcited by 13
  26. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 13
  27. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 12
  28. 2024Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localizationcited by 10
  29. 2023Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hearcited by 9
  30. 2022What do navigation agents learn about their environment?cited by 8
  31. 2023Chat2Map: Efficient Scene Mapping from Multi-Ego Conversationscited by 8
  32. 2024Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Predictioncited by 7
  33. 2021V-SlowFast Network for Efficient Visual Sound Separationcited by 6
  34. 2023Multi-goal Audio-visual Navigation using Sound Direction Mapcited by 6
  35. 2021Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Soundscited by 4
  36. 2022A Deep Reinforcement Learning Blind AI in DareFightingICEcited by 4
  37. 2022Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Visioncited by 4
  38. 2022Knowledge-driven Scene Priors for Semantic Audio-Visual Embodied Navigationcited by 4

50 most cited of 82 All 82 (JSON)