# ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data

arXiv: https://arxiv.org/abs/2107.04954 · HTML: https://mlsanity.com/paper/2107.04954/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 11 Jul 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [Learning Features of Music from Scratch](https://mlsanity.com/paper/1611.09827/) (2016)
- [Toward Interpretable Music Tagging with Self-Attention](https://mlsanity.com/paper/1906.04972/) (2019)
- [Adversarial Sampling and Training for Semi-Supervised Information Retrieval](https://mlsanity.com/paper/1811.04155/) (2018)
- [Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions](https://mlsanity.com/paper/2002.00212/) (2020)
- [Deep Polyphonic ADSR Piano Note Transcription](https://mlsanity.com/paper/1906.09165/) (2019)
- [Adversarial Learning for Improved Onsets and Frames Music Transcription](https://mlsanity.com/paper/1906.08512/) (2019)
- [Invariances and Data Augmentation for Supervised Music Transcription](https://mlsanity.com/paper/1711.04845/) (2017)
- [Musical Composition Style Transfer via Disentangled Timbre Representations](https://mlsanity.com/paper/1905.13567/) (2019)
- [A holistic approach to polyphonic music transcription with neural networks](https://mlsanity.com/paper/1910.12086/) (2019)
- [The Effect of Spectrogram Reconstruction on Automatic Music Transcription: An Alternative Approach to Improve Transcription Accuracy](https://mlsanity.com/paper/2010.09969/) (2020)
- [Large-Scale MIDI-based Composer Classification](https://mlsanity.com/paper/2010.14805/) (2020)
- [Revisiting the Onsets and Frames Model with Additive Attention](https://mlsanity.com/paper/2104.06607/) (2021)
- [Deep Unsupervised Drum Transcription](https://mlsanity.com/paper/1906.03697/) (2019)
- [Cosine-Distance Virtual Adversarial Training for Semi-Supervised Speaker-Discriminative Acoustic Embeddings](https://mlsanity.com/paper/2008.03756/) (2020)

## Built on by (most cited first, 20 of 20)

- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model](https://mlsanity.com/paper/2311.00968/) (2023)
- [Unaligned Supervision For Automatic Music Transcription in The Wild](https://mlsanity.com/paper/2204.13668/) (2022)
- [DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability](https://mlsanity.com/paper/2210.05148/) (2022)
- [Multitrack Music Transcription with a Time-Frequency Perceiver](https://mlsanity.com/paper/2306.10785/) (2023)
- [YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation](https://mlsanity.com/paper/2407.04822/) (2024)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [Music Foundation Model as Generic Booster for Music Downstream Tasks](https://mlsanity.com/paper/2411.01135/) (2024)
- [Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio](https://mlsanity.com/paper/2505.12863/) (2025)
- [Toward Fully Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2402.15569/) (2024)
- [Towards Efficient and Real-Time Piano Transcription Using Neural Autoregressive Models](https://mlsanity.com/paper/2404.06818/) (2024)
- [Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training](https://mlsanity.com/paper/2302.00286/) (2023)
- [Timbre-Trap: A Low-Resource Framework for Instrument-Agnostic Music Transcription](https://mlsanity.com/paper/2309.15717/) (2023)
- [Annotation-free Automatic Music Transcription with Scalable Synthetic Data and Adversarial Domain Confusion](https://mlsanity.com/paper/2312.10402/) (2023)
- [MR-MT3: Memory Retaining Multi-Track Music Transcription to Mitigate Instrument Leakage](https://mlsanity.com/paper/2403.10024/) (2024)
- [Machine Learning Techniques in Automatic Music Transcription: A Systematic Survey](https://mlsanity.com/paper/2406.15249/) (2024)
- [Deep-Learning Architectures for Multi-Pitch Estimation: Towards Reliable Evaluation](https://mlsanity.com/paper/2202.09198/) (2022)
- [Text2Data: Low-Resource Data Generation with Textual Control](https://mlsanity.com/paper/2402.10941/) (2024)
- [Music Transcription with (Almost) No Supervision](https://mlsanity.com/paper/2605.24193/) (2026)
- [Transfer of knowledge among instruments in automatic music transcription](https://mlsanity.com/paper/2305.00426/) (2023)

## Abstract (arXiv)

Most of the current supervised automatic music transcription (AMT) models lack the ability to generalize. This means that they have trouble transcribing real-world music recordings from diverse musical genres that are not presented in the labelled training data. In this paper, we propose a semi-supervised framework, ReconVAT, which solves this issue by leveraging the huge amount of available unlabelled music recordings. The proposed ReconVAT uses reconstruction loss and virtual adversarial training. When combined with existing U-net models for AMT, ReconVAT achieves competitive results on common benchmark datasets such as MAPS and MusicNet. For example, in the few-shot setting for the string part version of MusicNet, ReconVAT achieves F1-scores of 61.0% and 41.6% for the note-wise and note-with-offset-wise metrics respectively, which translates into an improvement of 22.2% and 62.5% compared to the supervised baseline model. Our proposed framework also demonstrates the potential of continual learning on new data, which could be useful in real-world applications whereby new data is constantly available.