# Mesostructures: Beyond Spectrogram Loss in Differentiable Time-Frequency Analysis

arXiv: https://arxiv.org/abs/2301.10183 · HTML: https://mlsanity.com/paper/2301.10183/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 20 references that are in the map.
- First posted on arXiv 24 Jan 2023.

## Builds on (most cited first, 6 of 6)

- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Kymatio: Scattering Transforms in Python](https://mlsanity.com/paper/1812.11214/) (2018)
- [I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch](https://mlsanity.com/paper/2012.04572/) (2020)
- [Joint Time-Frequency Scattering](https://mlsanity.com/paper/1807.08869/) (2018)
- [The Shape of RemiXXXes to Come: Audio Texture Synthesis with Time-frequency Scattering](https://mlsanity.com/paper/1906.09334/) (2019)

## Built on by (most cited first, 2 of 2)

- [SCRAPL: Scattering Transform with Random Paths for Machine Learning](https://mlsanity.com/paper/2602.11145/) (2026)
- [Evaluating Loss Functions in Differentiable Out-of-Domain Sound-Matching with Partial Parameter Distance](https://mlsanity.com/paper/2608.27698/) (2026)

## Abstract (arXiv)

Computer musicians refer to mesostructures as the intermediate levels of articulation between the microstructure of waveshapes and the macrostructure of musical forms. Examples of mesostructures include melody, arpeggios, syncopation, polyphonic grouping, and textural contrast. Despite their central role in musical expression, they have received limited attention in deep learning. Currently, autoencoders and neural audio synthesizers are only trained and evaluated at the scale of microstructure: i.e., local amplitude variations up to 100 milliseconds or so. In this paper, we formulate and address the problem of mesostructural audio modeling via a composition of a differentiable arpeggiator and time-frequency scattering. We empirically demonstrate that time--frequency scattering serves as a differentiable model of similarity between synthesis parameters that govern mesostructure. By exposing the sensitivity of short-time spectral distances to time alignment, we motivate the need for a time-invariant and multiscale differentiable time--frequency model of similarity at the level of both local spectra and spectrotemporal modulations.