# Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms

arXiv: https://arxiv.org/abs/2105.05791 · HTML: https://mlsanity.com/paper/2105.05791/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 68 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Transformers without Tears: Improving the Normalization of Self-Attention](https://mlsanity.com/paper/1910.05895/) (2019)
- [Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity](https://mlsanity.com/paper/1909.08494/) (2019)
- [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014)
- [madmom: a new Python Audio and Music Signal Processing Library](https://mlsanity.com/paper/1605.07008/) (2016)
- [Distilling the Knowledge of BERT for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2008.03822/) (2020)
- [Relative Positional Encoding for Speech Recognition and Direct Translation](https://mlsanity.com/paper/2005.09940/) (2020)
- [Data Augmentation for End-to-end Code-switching Speech Recognition](https://mlsanity.com/paper/2011.02160/) (2020)
- [Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition](https://mlsanity.com/paper/1907.06017/) (2019)
- [Learning Binary Residual Representations for Domain-specific Video Streaming](https://mlsanity.com/paper/1712.05087/) (2017)
- [Distilling Knowledge Learned in BERT for Text Generation](https://mlsanity.com/paper/1911.03829/) (2019)
- [Improved Chord Recognition by Combining Duration and Harmonic Language Models](https://mlsanity.com/paper/1808.05335/) (2018)
- [On the Futility of Learning Complex Frame-Level Language Models for Chord Recognition](https://mlsanity.com/paper/1702.00178/) (2017)
- [Automatic Chord Recognition with Higher-Order Harmonic Language Modelling](https://mlsanity.com/paper/1808.05341/) (2018)
- [Tatum-Level Drum Transcription Based on a Convolutional Recurrent Neural Network with Language Model-Based Regularized Training](https://mlsanity.com/paper/2010.03749/) (2020)

## Built on by (most cited first, 1 of 1)

- [Toward Deep Drum Source Separation](https://mlsanity.com/paper/2312.09663/) (2023)

## Abstract (arXiv)

This paper describes an automatic drum transcription (ADT) method that directly estimates a tatum-level drum score from a music signal, in contrast to most conventional ADT methods that estimate the frame-level onset probabilities of drums. To estimate a tatum-level score, we propose a deep transcription model that consists of a frame-level encoder for extracting the latent features from a music signal and a tatum-level decoder for estimating a drum score from the latent features pooled at the tatum level. To capture the global repetitive structure of drum scores, which is difficult to learn with a recurrent neural network (RNN), we introduce a self-attention mechanism with tatum-synchronous positional encoding into the decoder. To mitigate the difficulty of training the self-attention-based model from an insufficient amount of paired data and improve the musical naturalness of the estimated scores, we propose a regularized training method that uses a global structure-aware masked language (score) model with a self-attention mechanism pretrained from an extensive collection of drum scores. Experimental results showed that the proposed regularized model outperformed the conventional RNN-based model in terms of the tatum-level error rate and the frame-level F-measure, even when only a limited amount of paired data was available so that the non-regularized model underperformed the RNN-based model.