mlsanityData to 14 Oct 2026

Structured-Noise Masked Modeling for Video, Audio and Beyond

arXiv 2503.16311 ↗First posted 20 Mar 2025cs.LG cs.AI cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

Masked modeling has emerged as a robust self-supervised learning framework. However, most methods rely on random masking, which disregards the structural properties of different data modalities. To align with the spatiotemporal and spectral characteristics of video and audio data, we introduce a structured noise-based masking approach. By filtering white noise into different color noise distributions, we generate structured masks that capture modality-specific patterns without requiring handcrafted heuristics or access to the data. Our approach enhances masked video and audio modeling frameworks without any additional computational cost. Experiments show that structured noise masking consistently outperforms random masking, underscoring the value of modality-aware masking strategies for representation learning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 63 of its 77 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

63 of 77 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  10. 2017The Kinetics Human Action Video Datasetcited by 1,909
  11. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  12. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
38 more
  1. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  2. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  3. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  4. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  5. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  6. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  9. 2021AST: Audio Spectrogram Transformercited by 643
  10. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  11. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  12. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  13. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 383
  14. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  15. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 344
  16. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  17. 2022Masked Autoencoders that Listencited by 273
  18. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 234
  19. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  20. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  21. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  22. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  23. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  24. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 114
  25. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  26. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  27. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  28. 2022Masked Frequency Modeling for Self-Supervised Visual Pre-Trainingcited by 71
  29. 2022MAViL: Masked Audio-Video Learnerscited by 59
  30. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 51
  31. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 43
  32. 2023MGMAE: Motion Guided Masking for Video Masked Autoencodingcited by 41
  33. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  34. 2023Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representationscited by 35
  35. 2020Context-aware and Scale-insensitive Temporal Repetition Countingcited by 34
  36. 2022mc-BEiT: Multi-choice Discretization for Image BERT Pre-trainingcited by 31
  37. 2023Motion-Guided Masking for Spatiotemporal Representation Learningcited by 30
  38. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026JEPA Learns What the Mask Leaves Unrecoverablecited by 0