# Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Framework

arXiv: https://arxiv.org/abs/2608.09529 · HTML: https://mlsanity.com/paper/2608.09529/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 64 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 30 of 32)

- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack](https://mlsanity.com/paper/2309.15807/) (2023)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Any-to-Any Generation via Composable Diffusion](https://mlsanity.com/paper/2305.11846/) (2023)
- [TransNet V2: An effective deep network architecture for fast shot transition detection](https://mlsanity.com/paper/2008.04838/) (2020)
- [Visually Indicated Sounds](https://mlsanity.com/paper/1512.08512/) (2015)
- [Instance-Conditioned GAN](https://mlsanity.com/paper/2109.05070/) (2021)
- [Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation](https://mlsanity.com/paper/2508.09987/) (2025)
- [Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning](https://mlsanity.com/paper/2308.11276/) (2023)
- [Speech Recognition with Augmented Synthesized Speech](https://mlsanity.com/paper/1909.11699/) (2019)
- [AI Flow at the Network Edge](https://mlsanity.com/paper/2411.12469/) (2024)
- [Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment](https://mlsanity.com/paper/2303.17490/) (2023)
- [SAM Audio: Segment Anything in Audio](https://mlsanity.com/paper/2512.18099/) (2025)
- [Sound2Sight: Generating Visual Dynamics from Sound and Context](https://mlsanity.com/paper/2007.12130/) (2020)
- [GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation](https://mlsanity.com/paper/2303.10056/) (2023)
- [Learning Visual Styles from Audio-Visual Associations](https://mlsanity.com/paper/2205.05072/) (2022)
- [AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation](https://mlsanity.com/paper/2305.13050/) (2023)
- [NFIG: Multi-Scale Autoregressive Image Generation via Frequency Ordering](https://mlsanity.com/paper/2503.07076/) (2025)
- [VGGSounder: Audio-Visual Evaluations for Foundation Models](https://mlsanity.com/paper/2508.08237/) (2025)
- [RealSyn: An Effective and Scalable Multimodal Interleaved Document Transformation Paradigm](https://mlsanity.com/paper/2502.12513/) (2025)
- [FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion](https://mlsanity.com/paper/2506.01111/) (2025)
- [Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation](https://mlsanity.com/paper/2605.08729/) (2026)

## Abstract (arXiv)

As an important subfield of cross-modal generation, synthesizing static visual content in the form of images from audio, namely audio-to-image (A2I) generation, has attracted increasing research attention in recent years. Nevertheless, despite the remarkable visual quality of modern text-to-image (T2I) models, the performance of A2I remains fundamentally limited by traditional datasets, which often lack both high-fidelity images and precise cross-modal alignment. As a result, existing methods still struggle to achieve high-quality audio-to-image generation through finetuning strong T2I models, thereby constraining practical applications in this area. Motivated by this gap, we introduce A2I-Set, a unified, high-quality tri-modal dataset consisting of 323K paired audio, images, and detailed text captions, specifically designed for audio-visual research, including audio-conditioned image generation. Besides, we developed a new mixed-source test set for the A2I task through human supervision. We further propose an A2I model, AudioCanvas, fine-tuned on our A2I-Set. Experiments show that AudioCanvas achieves more visually expressive as well as cross-modal alignment results that generally outperforming existing approaches. Our dataset and source code are available at https://github.com/gdx012/A2I-Generation.