# Network Bending of Diffusion Models for Audio-Visual Generation

arXiv: https://arxiv.org/abs/2406.19589 · HTML: https://mlsanity.com/paper/2406.19589/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 15 of 15)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Kornia: an Open Source Differentiable Computer Vision Library for PyTorch](https://mlsanity.com/paper/1910.02190/) (2019)
- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Rewriting a Deep Generative Model](https://mlsanity.com/paper/2007.15646/) (2020)
- [Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder](https://mlsanity.com/paper/2403.10255/) (2024)
- [AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation](https://mlsanity.com/paper/2305.13050/) (2023)
- [Generative Disco: Text-to-Video Generation for Music Visualization](https://mlsanity.com/paper/2304.08551/) (2023)
- [Tr"aumerAI: Dreaming Music with StyleGAN](https://mlsanity.com/paper/2102.04680/) (2021)
- [Crossing You in Style: Cross-modal Style Transfer from Music to Visual Arts](https://mlsanity.com/paper/2009.08083/) (2020)
- [NAViDAd: A No-Reference Audio-Visual Quality Metric Based on a Deep Autoencoder](https://mlsanity.com/paper/2001.11406/) (2020)
- [A Preliminary Model for the Design of Music Visualizations](https://mlsanity.com/paper/2104.04922/) (2021)

## Built on by (most cited first, 3 of 3)

- [Explainability-in-Action: Enabling Expressive Manipulation and Tacit Understanding by Bending Diffusion Models in ComfyUI](https://mlsanity.com/paper/2508.07183/) (2025)
- [AttentionBender: Manipulating Cross-Attention in Video Diffusion Transformers as a Creative Probe](https://mlsanity.com/paper/2604.20936/) (2026)
- [Unboxing Diffusion Models for the Arts: Interactive Model Bending and Practice-Based Explainability](https://mlsanity.com/paper/2607.22428/) (2026)

## Abstract (arXiv)

In this paper we present the first steps towards the creation of a tool which enables artists to create music visualizations using pre-trained, generative, machine learning models. First, we investigate the application of network bending, the process of applying transforms within the layers of a generative network, to image generation diffusion models by utilizing a range of point-wise, tensor-wise, and morphological operators. We identify a number of visual effects that result from various operators, including some that are not easily recreated with standard image editing tools. We find that this process allows for continuous, fine-grain control of image generation which can be helpful for creative applications. Next, we generate music-reactive videos using Stable Diffusion by passing audio features as parameters to network bending operators. Finally, we comment on certain transforms which radically shift the image and the possibilities of learning more about the latent space of Stable Diffusion based on these transforms.