# SeeingSounds: Learning Audio-to-Visual Alignment via Text

arXiv: https://arxiv.org/abs/2510.11738 · HTML: https://mlsanity.com/paper/2510.11738/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Any-to-Any Generation via Composable Diffusion](https://mlsanity.com/paper/2305.11846/) (2023)
- [Visual to Sound: Generating Natural Sound for Videos in the Wild](https://mlsanity.com/paper/1712.01393/) (2017)
- [Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation](https://mlsanity.com/paper/2309.16429/) (2023)
- [Deep Cross-Modal Audio-Visual Generation](https://mlsanity.com/paper/1704.08292/) (2017)
- [Prompt Design and Engineering: Introduction and Advanced Methods](https://mlsanity.com/paper/2401.14423/) (2024)
- [Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment](https://mlsanity.com/paper/2303.17490/) (2023)
- [CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation](https://mlsanity.com/paper/1711.08102/) (2017)
- [GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation](https://mlsanity.com/paper/2303.10056/) (2023)
- [Towards Audio to Scene Image Synthesis using Generative Adversarial Network](https://mlsanity.com/paper/1808.04108/) (2018)
- [Learning Visual Styles from Audio-Visual Associations](https://mlsanity.com/paper/2205.05072/) (2022)
- [SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models](https://mlsanity.com/paper/2405.00878/) (2024)
- [Revealing Vision-Language Integration in the Brain with Multimodal Networks](https://mlsanity.com/paper/2406.14481/) (2024)
- [Self-Attentive Pooling for Efficient Deep Learning](https://mlsanity.com/paper/2209.07659/) (2022)
- [Estimating Visual Information From Audio Through Manifold Learning](https://mlsanity.com/paper/2208.02337/) (2022)

## Abstract (arXiv)

We introduce SeeingSounds, a lightweight and modular framework for audio-to-image generation that leverages the interplay between audio, language, and vision-without requiring any paired audio-visual data or training on visual generative models. Rather than treating audio as a substitute for text or relying solely on audio-to-text mappings, our method performs dual alignment: audio is projected into a semantic language space via a frozen language encoder, and, contextually grounded into the visual domain using a vision-language model. This approach, inspired by cognitive neuroscience, reflects the natural cross-modal associations observed in human perception. The model operates on frozen diffusion backbones and trains only lightweight adapters, enabling efficient and scalable learning. Moreover, it supports fine-grained and interpretable control through procedural text prompt generation, where audio transformations (e.g., volume or pitch shifts) translate into descriptive prompts (e.g., "a distant thunder") that guide visual outputs. Extensive experiments across standard benchmarks confirm that SeeingSounds outperforms existing methods in both zero-shot and supervised settings, establishing a new state of the art in controllable audio-to-visual generation.