# The Effect of Spectrogram Reconstruction on Automatic Music Transcription: An Alternative Approach to Improve Transcription Accuracy

arXiv: https://arxiv.org/abs/2010.09969 · HTML: https://mlsanity.com/paper/2010.09969/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 10 of 10)

- [Automatic Piano Transcription with Hierarchical Frequency-Time Transformer](https://mlsanity.com/paper/2307.04305/) (2023)
- [ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data](https://mlsanity.com/paper/2107.04954/) (2021)
- [DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability](https://mlsanity.com/paper/2210.05148/) (2022)
- [Toward Fully Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2402.15569/) (2024)
- [STAB: Speech Tokenizer Assessment Benchmark](https://mlsanity.com/paper/2409.02384/) (2024)
- [Timbre-Trap: A Low-Resource Framework for Instrument-Agnostic Music Transcription](https://mlsanity.com/paper/2309.15717/) (2023)
- [The Inverse Drum Machine: Source Separation Through Joint Transcription and Analysis-by-Synthesis](https://mlsanity.com/paper/2505.03337/) (2025)
- [Machine Learning Techniques in Automatic Music Transcription: A Systematic Survey](https://mlsanity.com/paper/2406.15249/) (2024)
- [Deep-Learning Architectures for Multi-Pitch Estimation: Towards Reliable Evaluation](https://mlsanity.com/paper/2202.09198/) (2022)
- [Transfer of knowledge among instruments in automatic music transcription](https://mlsanity.com/paper/2305.00426/) (2023)

## Abstract (arXiv)

Most of the state-of-the-art automatic music transcription (AMT) models break down the main transcription task into sub-tasks such as onset prediction and offset prediction and train them with onset and offset labels. These predictions are then concatenated together and used as the input to train another model with the pitch labels to obtain the final transcription. We attempt to use only the pitch labels (together with spectrogram reconstruction loss) and explore how far this model can go without introducing supervised sub-tasks. In this paper, we do not aim at achieving state-of-the-art transcription accuracy, instead, we explore the effect that spectrogram reconstruction has on our AMT model. Our proposed model consists of two U-nets: the first U-net transcribes the spectrogram into a posteriorgram, and a second U-net transforms the posteriorgram back into a spectrogram. A reconstruction loss is applied between the original spectrogram and the reconstructed spectrogram to constrain the second U-net to focus only on reconstruction. We train our model on three different datasets: MAPS, MAESTRO, and MusicNet. Our experiments show that adding the reconstruction loss can generally improve the note-level transcription accuracy when compared to the same model without the reconstruction part. Moreover, it can also boost the frame-level precision to be higher than the state-of-the-art models. The feature maps learned by our U-net contain gridlike structures (not present in the baseline model) which implies that with the presence of the reconstruction loss, the model is probably trying to count along both the time and frequency axis, resulting in a higher note-level transcription accuracy.