# Sound-VECaps: Improving Audio Generation with Visual Enhanced Captions

arXiv: https://arxiv.org/abs/2407.04416 · HTML: https://mlsanity.com/paper/2407.04416/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2024.

## Built on by (most cited first, 15 of 15)

- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows](https://mlsanity.com/paper/2508.06098/) (2025)
- [Omni2Sound: Towards Unified Video-Text-to-Audio Generation](https://mlsanity.com/paper/2601.02731/) (2026)
- [GLAP: General contrastive audio-text pretraining across domains and languages](https://mlsanity.com/paper/2506.11350/) (2025)
- [MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks](https://mlsanity.com/paper/2507.23511/) (2025)
- [Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos](https://mlsanity.com/paper/2507.11967/) (2025)
- [DreamAudio: Customized Text-to-Audio Generation with Diffusion Models](https://mlsanity.com/paper/2509.06027/) (2025)
- [Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation](https://mlsanity.com/paper/2506.08457/) (2025)
- [FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation](https://mlsanity.com/paper/2608.22111/) (2026)
- [Unify Variables in Neural Scaling Laws for General Audio Representations via Embedding Effective Rank](https://mlsanity.com/paper/2510.10948/) (2025)
- [ConceptCaps: a Distilled Concept Dataset for Interpretability in Music Models](https://mlsanity.com/paper/2601.14157/) (2026)
- [COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings](https://mlsanity.com/paper/2605.29628/) (2026)
- [Omni-Interactive Universal Embedder](https://mlsanity.com/paper/2608.27044/) (2026)
- [SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval](https://mlsanity.com/paper/2609.02343/) (2026)
- [AVSD-Scenes: A Dataset for Audio-Visual Description of Urban Scenes](https://mlsanity.com/paper/2610.01861/) (2026)

## Abstract (arXiv)

Generative models have shown significant achievements in audio generation tasks. However, existing models struggle with complex and detailed prompts, leading to potential performance degradation. We hypothesize that this problem stems from the simplicity and scarcity of the training data. This work aims to create a large-scale audio dataset with rich captions for improving audio generation models. We first develop an automated pipeline to generate detailed captions by transforming predicted visual captions, audio captions, and tagging labels into comprehensive descriptions using a Large Language Model (LLM). The resulting dataset, Sound-VECaps, comprises 1.66M high-quality audio-caption pairs with enriched details including audio event orders, occurred places and environment information. We then demonstrate that training the text-to-audio generation models with Sound-VECaps significantly improves the performance on complex prompts. Furthermore, we conduct ablation studies of the models on several downstream audio-language tasks, showing the potential of Sound-VECaps in advancing audio-text representation learning. Our dataset and models are available online from here https://yyua8222.github.io/Sound-VECaps-demo/.