# Masked Modeling Duo: Towards a Universal Audio Pre-training Framework

arXiv: https://arxiv.org/abs/2404.06095 · HTML: https://mlsanity.com/paper/2404.06095/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2024.

## Built on by (most cited first, 23 of 23)

- [SONICS: Synthetic Or Not -- Identifying Counterfeit Songs](https://mlsanity.com/paper/2408.14080/) (2024)
- [Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning](https://mlsanity.com/paper/2507.02915/) (2025)
- [Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importance](https://mlsanity.com/paper/2504.01690/) (2025)
- [Towards a Unified Representation Evaluation Framework Beyond Downstream Tasks](https://mlsanity.com/paper/2505.06224/) (2025)
- [MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning](https://mlsanity.com/paper/2508.12709/) (2025)
- [Myna: Masking-Based Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2502.12511/) (2025)
- [Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles](https://mlsanity.com/paper/2505.22027/) (2025)
- [Lungmix: A Mixup-Based Strategy for Generalization in Respiratory Sound Classification](https://mlsanity.com/paper/2501.00064/) (2024)
- [Emergent musical properties of a transformer under contrastive self-supervised learning](https://mlsanity.com/paper/2506.23873/) (2025)
- [UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation](https://mlsanity.com/paper/2602.21772/) (2026)
- [Performance improvement of spatial semantic segmentation with enriched audio features and agent-based error correction for DCASE 2025 Challenge Task 4](https://mlsanity.com/paper/2506.21174/) (2025)
- [Improving Deep Learning-based Respiratory Sound Analysis with Frequency Selection and Attention Mechanism](https://mlsanity.com/paper/2507.20052/) (2025)
- [Representation-Based Data Quality Audits for Audio](https://mlsanity.com/paper/2509.26291/) (2025)
- [AaSP: Aliasing-aware Self-Supervised Pre-Training for Audio Spectrogram Transformers](https://mlsanity.com/paper/2512.03637/) (2025)
- [Adaptive Multimodal Person Recognition: A Robust Framework for Handling Missing Modalities](https://mlsanity.com/paper/2512.14961/) (2025)
- [Geometry-Aware Optimization for Respiratory Sound Classification: Enhancing Sensitivity with SAM-Optimized Audio Spectrogram Transformers](https://mlsanity.com/paper/2512.22564/) (2025)
- [Representation-Regularized Convolutional Audio Transformer for Audio Understanding](https://mlsanity.com/paper/2601.21612/) (2026)
- [Sub-Band Spectral Matching with Localized Score Aggregation for Robust Anomalous Sound Detection](https://mlsanity.com/paper/2603.13749/) (2026)
- [Meta-Ensemble Learning with Diverse Data Splits for Improved Respiratory Sound Classification](https://mlsanity.com/paper/2604.24096/) (2026)
- [Echo: A Joint-Embedding Predictive Architecture for Speaker Diarization and Speech Recognition in a Shared Latent Space](https://mlsanity.com/paper/2606.01909/) (2026)
- [Lung-SRAD: Spectral-Aware Regularized Audio DASS with Dual-Axis Patch-Mix Contrastive Learning for Respiratory Sound Classification](https://mlsanity.com/paper/2606.11922/) (2026)
- [BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations](https://mlsanity.com/paper/2606.30700/) (2026)
- [Device Invariance using Domain Adaptation on Acoustic Scene Classification](https://mlsanity.com/paper/2607.25887/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) using masked prediction has made great strides in general-purpose audio representation. This study proposes Masked Modeling Duo (M2D), an improved masked prediction SSL, which learns by predicting representations of masked input signals that serve as training signals. Unlike conventional methods, M2D obtains a training signal by encoding only the masked part, encouraging the two networks in M2D to model the input. While M2D improves general-purpose audio representations, a specialized representation is essential for real-world applications, such as in industrial and medical domains. The often confidential and proprietary data in such domains is typically limited in size and has a different distribution from that in pre-training datasets. Therefore, we propose M2D for X (M2D-X), which extends M2D to enable the pre-training of specialized representations for an application X. M2D-X learns from M2D and an additional task and inputs background noise. We make the additional task configurable to serve diverse applications, while the background noise helps learn on small data and forms a denoising task that makes representation robust. With these design choices, M2D-X should learn a representation specialized to serve various application needs. Our experiments confirmed that the representations for general-purpose audio, specialized for the highly competitive AudioSet and speech domain, and a small-data medical task achieve top-level performance, demonstrating the potential of using our models as a universal audio pre-training framework. Our code is available online for future studies at https://github.com/nttcslab/m2d