# Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentation

arXiv: https://arxiv.org/abs/2309.17352 · HTML: https://mlsanity.com/paper/2309.17352/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2023.

## Built on by (most cited first, 19 of 19)

- [Large Language Models are Efficient Learners of Noise-Robust Speech Recognition](https://mlsanity.com/paper/2401.10446/) (2024)
- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models](https://mlsanity.com/paper/2505.17568/) (2025)
- [Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding](https://mlsanity.com/paper/2406.13275/) (2024)
- [DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning](https://mlsanity.com/paper/2410.09472/) (2024)
- [Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition](https://mlsanity.com/paper/2408.09215/) (2024)
- [Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairness](https://mlsanity.com/paper/2409.10825/) (2024)
- [KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models](https://mlsanity.com/paper/2409.11057/) (2024)
- [TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages](https://mlsanity.com/paper/2402.16021/) (2024)
- [EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance](https://mlsanity.com/paper/2409.01201/) (2024)
- [CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer](https://mlsanity.com/paper/2506.00800/) (2025)
- [HearSay Benchmark: Do Audio LLMs Leak What They Hear?](https://mlsanity.com/paper/2601.03783/) (2026)
- [Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning](https://mlsanity.com/paper/2409.01160/) (2024)
- [EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generation](https://mlsanity.com/paper/2410.12028/) (2024)
- [AISTAT lab system for DCASE2025 Task6: Language-based audio retrieval](https://mlsanity.com/paper/2509.16649/) (2025)
- [LLM-Driven Completeness and Consistency Evaluation for Cultural Heritage Data Augmentation in Cross-Modal Retrieval](https://mlsanity.com/paper/2511.06268/) (2025)
- [Semantic-Aware Confidence Calibration for Automated Audio Captioning](https://mlsanity.com/paper/2512.10170/) (2025)
- [Cross-Linguistic Persona-Driven Data Synthesis for Robust Multimodal Cognitive Decline Detection](https://mlsanity.com/paper/2602.07978/) (2026)

## Abstract (arXiv)

Automated audio captioning (AAC) aims to generate informative descriptions for various sounds from nature and/or human activities. In recent years, AAC has quickly attracted research interest, with state-of-the-art systems now relying on a sequence-to-sequence (seq2seq) backbone powered by strong models such as Transformers. Following the macro-trend of applied machine learning research, in this work, we strive to improve the performance of seq2seq AAC models by extensively leveraging pretrained models and large language models (LLMs). Specifically, we utilize BEATs to extract fine-grained audio features. Then, we employ Instructor LLM to fetch text embeddings of captions, and infuse their language-modality knowledge into BEATs audio features via an auxiliary InfoNCE loss function. Moreover, we propose a novel data augmentation method that uses ChatGPT to produce caption mix-ups (i.e., grammatical and compact combinations of two captions) which, together with the corresponding audio mixtures, increase not only the amount but also the complexity and diversity of training data. During inference, we propose to employ nucleus sampling and a hybrid reranking algorithm, which has not been explored in AAC research. Combining our efforts, our model achieves a new state-of-the-art 32.6 SPIDEr-FL score on the Clotho evaluation split, and wins the 2023 DCASE AAC challenge.