# ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing

arXiv: https://arxiv.org/abs/2506.21448 · HTML: https://mlsanity.com/paper/2506.21448/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 51 of its 62 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)
- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 51)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [An Edit Friendly DDPM Noise Space: Inversion and Manipulations](https://mlsanity.com/paper/2304.06140/) (2023)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)

## Built on by (most cited first, 30 of 34)

- [HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation](https://mlsanity.com/paper/2508.16930/) (2025)
- [VABench: A Comprehensive Benchmark for Audio-Video Generation](https://mlsanity.com/paper/2512.09299/) (2025)
- [UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist](https://mlsanity.com/paper/2511.08521/) (2025)
- [Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing](https://mlsanity.com/paper/2604.10708/) (2026)
- [Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction](https://mlsanity.com/paper/2510.03117/) (2025)
- [PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation](https://mlsanity.com/paper/2512.23994/) (2025)
- [Omni2Sound: Towards Unified Video-Text-to-Audio Generation](https://mlsanity.com/paper/2601.02731/) (2026)
- [PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation](https://mlsanity.com/paper/2511.18833/) (2025)
- [EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation](https://mlsanity.com/paper/2512.24731/) (2025)
- [SoundReactor: Frame-level Online Video-to-Audio Generation](https://mlsanity.com/paper/2510.02110/) (2025)
- [3MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation](https://mlsanity.com/paper/2511.21780/) (2025)
- [Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video](https://mlsanity.com/paper/2510.21581/) (2025)
- [AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer](https://mlsanity.com/paper/2603.15597/) (2026)
- [ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation](https://mlsanity.com/paper/2512.03036/) (2025)
- [Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks](https://mlsanity.com/paper/2602.01630/) (2026)
- [V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation](https://mlsanity.com/paper/2603.11042/) (2026)
- [FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts](https://mlsanity.com/paper/2603.19857/) (2026)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [WavFlow: Audio Generation in Waveform Space](https://mlsanity.com/paper/2605.18749/) (2026)
- [Benchmarking Single-Factor Physical Video-to-Audio Generation](https://mlsanity.com/paper/2605.30339/) (2026)
- [SightSound-R1: Cross-Modal Reasoning Distillation from Vision to Audio Language Models](https://mlsanity.com/paper/2509.15661/) (2025)
- [Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generation](https://mlsanity.com/paper/2510.08078/) (2025)
- [AVFakeBench: A Comprehensive Audio-Video Forgery Detection Benchmark for AV-LMMs](https://mlsanity.com/paper/2511.21251/) (2025)
- [Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition](https://mlsanity.com/paper/2603.12046/) (2026)
- [Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer](https://mlsanity.com/paper/2605.30940/) (2026)
- [Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models](https://mlsanity.com/paper/2607.09091/) (2026)
- [MultiSoundGen: Video-to-Audio Generation for Multi-Event Scenarios via SlowFast Contrastive Audio-Visual Pretraining and Direct Preference Optimization](https://mlsanity.com/paper/2509.19999/) (2025)
- [ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment](https://mlsanity.com/paper/2605.30965/) (2026)
- [STAR-VAE: Structured Topology-Aware Regularization for Audio Reconstruction and Generation](https://mlsanity.com/paper/2606.23064/) (2026)
- [A Production-Oriented Framework for Evaluation of SFX Generation](https://mlsanity.com/paper/2607.09973/) (2026)

## Abstract (arXiv)

While end-to-end video-to-audio generation has greatly improved, producing high-fidelity audio that authentically captures the nuances of visual content remains challenging. Like professionals in the creative industries, this generation requires sophisticated reasoning about items such as visual dynamics, acoustic environments, and temporal relationships. We present ThinkSound, a novel framework that leverages Chain-of-Thought (CoT) reasoning to enable stepwise, interactive audio generation and editing for videos. Our approach decomposes the process into three complementary stages: foundational foley generation that creates semantically coherent soundscapes, interactive object-centric refinement through precise user interactions, and targeted editing guided by natural language instructions. At each stage, a multimodal large language model generates contextually aligned CoT reasoning that guides a unified audio foundation model. Furthermore, we introduce AudioCoT, a comprehensive dataset with structured reasoning annotations that establishes connections between visual content, textual descriptions, and sound synthesis. Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics, and excels in the out-of-distribution Movie Gen Audio benchmark. The project page is available at https://ThinkSound-Project.github.io.