# The Mirrornet : Learning Audio Synthesizer Controls Inspired by Sensorimotor Interaction

arXiv: https://arxiv.org/abs/2110.05695 · HTML: https://mlsanity.com/paper/2110.05695/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 21 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 5 of 5)

- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Learning robust speech representation with an articulatory-regularized variational autoencoder](https://mlsanity.com/paper/2104.03204/) (2021)
- [Learning Joint Articulatory-Acoustic Representations with Normalizing Flows](https://mlsanity.com/paper/2005.09463/) (2020)

## Built on by (most cited first, 2 of 2)

- [Articulation GAN: Unsupervised modeling of articulatory learning](https://mlsanity.com/paper/2210.15173/) (2022)
- [CiwaGAN: Articulatory information exchange](https://mlsanity.com/paper/2309.07861/) (2023)

## Abstract (arXiv)

Experiments to understand the sensorimotor neural interactions in the human cortical speech system support the existence of a bidirectional flow of interactions between the auditory and motor regions. Their key function is to enable the brain to `learn' how to control the vocal tract for speech production. This idea is the impetus for the recently proposed "MirrorNet", a constrained autoencoder architecture. In this paper, the MirrorNet is applied to learn, in an unsupervised manner, the controls of a specific audio synthesizer (DIVA) to produce melodies only from their auditory spectrograms. The results demonstrate how the MirrorNet discovers the synthesizer parameters to generate the melodies that closely resemble the original and those of unseen melodies, and even determine the best set parameters to approximate renditions of complex piano melodies generated by a different synthesizer. This generalizability of the MirrorNet illustrates its potential to discover from sensory data the controls of arbitrary motor-plants.