mlsanityData to 7 Oct 2026

Segment Beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation

arXiv 2312.08673 ↗First posted 14 Dec 2023cs.CV cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Augmented Reality (AR) devices, emerging as prominent mobile interaction platforms, face challenges in user safety, particularly concerning oncoming vehicles. While some solutions leverage onboard camera arrays, these cameras often have limited field-of-view (FoV) with front or downward perspectives. Addressing this, we propose a new out-of-view semantic segmentation task and Segment Beyond View (SBV), a novel audio-visual semantic segmentation method. SBV supplements the visual modality, which miss the information beyond FoV, with the auditory information using a teacher-student distillation model (Omni2Ego). The model consists of a vision teacher utilising panoramic information, an auditory teacher with 8-channel audio, and an audio-visual student that takes views with limited FoV and binaural audio as input and produce semantic segmentation for objects outside FoV. SBV outperforms existing models in comparative evaluations and shows a consistent performance across varying FoV ranges and in monaural audio settings.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

22 of 45 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  7. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  8. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  9. 2021Localizing Visual Sounds the Hard Waycited by 169
  10. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 159
  11. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  12. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
10 more
  1. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  2. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
  3. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 54
  4. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  5. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 43
  6. 2020Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Soundscited by 26
  7. 2021Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Trackingcited by 9
  8. 2022Self-Supervised Moving Vehicle Detection from Audio-Visual Cuescited by 8
  9. 2021Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Soundscited by 4
  10. 2023An EEG-based Experiment on VR Sickness and Postural Instability While Walking in Virtual Environmentscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 4.

4 papers.

  1. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  2. 2024MMP: Towards Robust Multi-Modal Learning with Masked Modality Projectioncited by 10
  3. 2024Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modalitycited by 4
  4. 2024Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalitiescited by 0