# Exploring Sampling Techniques for Generating Melodies with a Transformer Language Model

arXiv: https://arxiv.org/abs/2308.09454 · HTML: https://mlsanity.com/paper/2308.09454/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 18 Aug 2023.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [C-RNN-GAN: Continuous recurrent neural networks with adversarial training](https://mlsanity.com/paper/1611.09904/) (2016)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation](https://mlsanity.com/paper/2210.10349/) (2022)
- [Music transcription modelling and composition using deep learning](https://mlsanity.com/paper/1604.08723/) (2016)
- [On the Typicality of Musical Sequences](https://mlsanity.com/paper/2211.13016/) (2022)

## Built on by (most cited first, 3 of 3)

- [Controlling Surprisal in Music Generation via Information Content Curve Matching](https://mlsanity.com/paper/2408.06022/) (2024)
- [Estimating Musical Surprisal in Audio](https://mlsanity.com/paper/2501.07474/) (2025)
- [Estimating Musical Surprisal from Audio in Autoregressive Diffusion Model Noise Spaces](https://mlsanity.com/paper/2508.05306/) (2025)

## Abstract (arXiv)

Research in natural language processing has demonstrated that the quality of generations from trained autoregressive language models is significantly influenced by the used sampling strategy. In this study, we investigate the impact of different sampling techniques on musical qualities such as diversity and structure. To accomplish this, we train a high-capacity transformer model on a vast collection of highly-structured Irish folk melodies and analyze the musical qualities of the samples generated using distribution truncation sampling techniques. Specifically, we use nucleus sampling, the recently proposed "typical sampling", and conventional ancestral sampling. We evaluate the effect of these sampling strategies in two scenarios: optimal circumstances with a well-calibrated model and suboptimal circumstances where we systematically degrade the model's performance. We assess the generated samples using objective and subjective evaluations. We discover that probability truncation techniques may restrict diversity and structural patterns in optimal circumstances, but may also produce more musical samples in suboptimal circumstances.