mlsanityData to 14 Oct 2026

Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixup

arXiv 2503.02284 ↗First posted 4 Mar 2025cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Video action recognition is a challenging but important task for understanding and discovering what the video does. However, acquiring annotations for a video is costly, and semi-supervised learning (SSL) has been studied to improve performance even with a small number of labeled data in the task. Prior studies for semi-supervised video action recognition have mostly focused on using single modality - visuals - but the video is multi-modal, so utilizing both visuals and audio would be desirable and improve performance further, which has not been explored well. Therefore, we propose audio-visual SSL for video action recognition, which uses both visual and audio together, even with quite a few labeled data, which is challenging. In addition, to maximize the information of audio and video, we propose a novel audio source localization-guided mixup method that considers inter-modal relations between video and audio modalities. In experiments on UCF-51, Kinetics-400, and VGGSound datasets, our model shows the superior performance of the proposed semi-supervised audio-visual action recognition framework and audio source localization-guided mixup.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 46 of its 57 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

46 of 57 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2014Going Deeper with Convolutionscited by 4,241
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  9. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  10. 2017The Kinetics Human Action Video Datasetcited by 1,909
  11. 2021ViViT: A Video Vision Transformercited by 1,620
  12. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
34 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  2. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  3. 2021Video Swin Transformercited by 1,091
  4. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,057
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  6. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  7. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  8. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  9. 2022FreeMatch: Self-adaptive Thresholding for Semi-supervised Learningcited by 232
  10. 2018Learning to Localize Sound Source in Visual Scenescited by 189
  11. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 148
  12. 2022Localizing Visual Sounds the Easy Waycited by 82
  13. 2022SVFormer: Semi-supervised Video Transformer for Action Recognitioncited by 68
  14. 2022Debiased Self-Training for Semi-Supervised Learningcited by 62
  15. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  16. 2021Semi-Supervised Action Recognition with Temporal Contrastive Learningcited by 50
  17. 2023Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learningcited by 47
  18. 2021MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognitioncited by 46
  19. 2021Multiview Pseudo-Labeling for Semi-supervised Learning from Videocited by 32
  20. 2021TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learningcited by 31
  21. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 31
  22. 2021Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognitioncited by 29
  23. 2020VideoSSL: Semi-Supervised Learning for Video Classificationcited by 27
  24. 2022Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognitioncited by 27
  25. 2019Efficient Video Classification Using Fewer Framescited by 26
  26. 2021Learning from Temporal Gradient for Semi-supervised Action Recognitioncited by 26
  27. 2023Boosting Semi-Supervised Learning by Exploiting All Unlabeled Datacited by 26
  28. 2021Barely-Supervised Learning: Semi-Supervised Learning with very few labeled imagescited by 17
  29. 2023TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognitioncited by 16
  30. 2022Modality Mixer for Multi-modal Action Recognitioncited by 9
  31. 2019Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Networkcited by 6
  32. 2023Audio-Visual Spatial Integration and Recursive Attention for Robust Sound Source Localizationcited by 5
  33. 2023MAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformerscited by 4
  34. 2024Efficient Selective Audio Masked Multimodal Bottleneck Transformer for Audio-Video Classificationcited by 1

Built on by

Not cited yet.