mlsanityData to 14 Oct 2026

Audio-visual Event Localization on Portrait Mode Short Videos

arXiv 2504.06884 ↗First posted 9 Apr 2025cs.MM cs.AI cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Audio-visual event localization (AVEL) plays a critical role in multimodal scene understanding. While existing datasets for AVEL predominantly comprise landscape-oriented long videos with clean and simple audio context, short videos have become the primary format of online video content due to the the proliferation of smartphones. Short videos are characterized by portrait-oriented framing and layered audio compositions (e.g., overlapping sound effects, voiceovers, and music), which brings unique challenges unaddressed by conventional methods. To this end, we introduce AVE-PM, the first AVEL dataset specifically designed for portrait mode short videos, comprising 25,335 clips that span 86 fine-grained categories with frame-level annotations. Beyond dataset creation, our empirical analysis shows that state-of-the-art AVEL methods suffer an average 18.66% performance drop during cross-mode evaluation. Further analysis reveals two key challenges of different video formats: 1) spatial bias from portrait-oriented framing introduces distinct domain priors, and 2) noisy audio composition compromise the reliability of audio modality. To address these issues, we investigate optimal preprocessing recipes and the impact of background music for AVEL on portrait mode videos. Experiments show that these methods can still benefit from tailored preprocessing and specialized model design, thus achieving improved performance. This work provides both a foundational benchmark and actionable insights for advancing AVEL research in the era of mobile-centric video content. Dataset and code will be released.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 23 of its 43 references that are in the map. First posted on arXiv 9 Apr 2025.

Builds on

23 of 43 references.

References · 23

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2018SlowFast Networks for Video Recognitioncited by 1,867
  4. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  5. 2021Multiscale Vision Transformerscited by 897
  6. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  7. 2017Look, Listen and Learncited by 458
  8. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  9. 2020Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervisioncited by 238
  10. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  11. 2020TDN: Temporal Difference Networks for Efficient Action Recognitioncited by 183
  12. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
11 more
  1. 2019Dual-modality seq2seq network for audio-visual event localizationcited by 72
  2. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  3. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  4. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  5. 2023Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baselinecited by 46
  6. 2021MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsingcited by 39
  7. 2024TIM: A Time Interval Machine for Audio-Visual Action Recognitioncited by 15
  8. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 14
  9. 2021MPN: Multimodal Parallel Network for Audio-Visual Event Localizationcited by 9
  10. 20223MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videoscited by 8
  11. 2023Video Recognition in Portrait Modecited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  2. 2025PreFM: Online Audio-Visual Event Parsing via Predictive Future Modelingcited by 4
  3. 2026Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentationcited by 2