# Towards Lightweight Controllable Audio Synthesis with Conditional Implicit Neural Representations

arXiv: https://arxiv.org/abs/2111.08462 · HTML: https://mlsanity.com/paper/2111.08462/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 14 Nov 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [Exploring the structure of a real-time, arbitrary neural artistic stylization network](https://mlsanity.com/paper/1705.06830/) (2017)
- [Generative Models as Distributions of Functions](https://mlsanity.com/paper/2102.04776/) (2021)
- [CDPAM: Contrastive learning for perceptual audio similarity](https://mlsanity.com/paper/2102.05109/) (2021)
- [I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch](https://mlsanity.com/paper/2012.04572/) (2020)
- [Dynamic Layer Normalization for Adaptive Neural Acoustic Modeling in Speech Recognition](https://mlsanity.com/paper/1707.06065/) (2017)
- [Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex](https://mlsanity.com/paper/1806.01845/) (2018)

## Built on by (most cited first, 5 of 5)

- [DISCO: Adversarial Defense with Local Implicit Functions](https://mlsanity.com/paper/2212.05630/) (2022)
- [Siamese SIREN: Audio Compression with Implicit Neural Representations](https://mlsanity.com/paper/2306.12957/) (2023)
- [Hypernetworks build Implicit Neural Representations of Sounds](https://mlsanity.com/paper/2302.04959/) (2023)
- [SPEAR: Receiver-to-Receiver Acoustic Neural Warping Field](https://mlsanity.com/paper/2406.11006/) (2024)
- [SegINR: Segment-wise Implicit Neural Representation for Sequence Alignment in Neural Text-to-Speech](https://mlsanity.com/paper/2410.04690/) (2024)

## Abstract (arXiv)

The high temporal resolution of audio and our perceptual sensitivity to small irregularities in waveforms make synthesizing at high sampling rates a complex and computationally intensive task, prohibiting real-time, controllable synthesis within many approaches. In this work we aim to shed light on the potential of Conditional Implicit Neural Representations (CINRs) as lightweight backbones in generative frameworks for audio synthesis. Our experiments show that small Periodic Conditional INRs (PCINRs) learn faster and generally produce quantitatively better audio reconstructions than Transposed Convolutional Neural Networks with equal parameter counts. However, their performance is very sensitive to activation scaling hyperparameters. When learning to represent more uniform sets, PCINRs tend to introduce artificial high-frequency components in reconstructions. We validate this noise can be minimized by applying standard weight regularization during training or decreasing the compositional depth of PCINRs, and suggest directions for future research.