# SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs

arXiv: https://arxiv.org/abs/2410.09503 · HTML: https://mlsanity.com/paper/2410.09503/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2024.

## Built on by (most cited first, 15 of 15)

- [MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix](https://mlsanity.com/paper/2505.13032/) (2025)
- [Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering](https://mlsanity.com/paper/2503.11197/) (2025)
- [Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model](https://mlsanity.com/paper/2501.07246/) (2025)
- [OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows](https://mlsanity.com/paper/2412.01169/) (2024)
- [SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing](https://mlsanity.com/paper/2601.09385/) (2026)
- [CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer](https://mlsanity.com/paper/2506.00800/) (2025)
- [Unleashing the Power of Natural Audio Featuring Multiple Sound Sources](https://mlsanity.com/paper/2504.17782/) (2025)
- [AC/DC: LLM-based Audio Comprehension via Dialogue Continuation](https://mlsanity.com/paper/2506.10312/) (2025)
- [LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2601.04658/) (2026)
- [TAD: Token-Adaptive Contrastive Decoding with Confidence-Guided Gating for Hallucination Mitigation in Large Audio-Language Models](https://mlsanity.com/paper/2609.07286/) (2026)
- [SAM: A Mamba-2 State-Space Audio-Language Model](https://mlsanity.com/paper/2509.15680/) (2025)
- [SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training](https://mlsanity.com/paper/2601.12594/) (2026)
- [Representation-Regularized Convolutional Audio Transformer for Audio Understanding](https://mlsanity.com/paper/2601.21612/) (2026)
- [CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning](https://mlsanity.com/paper/2607.04619/) (2026)
- [Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment](https://mlsanity.com/paper/2608.19825/) (2026)

## Abstract (arXiv)

Automated Audio Captioning (AAC) aims to generate natural textual descriptions for input audio signals. Recent progress in audio pre-trained models and large language models (LLMs) has significantly enhanced audio understanding and textual reasoning capabilities, making improvements in AAC possible. In this paper, we propose SLAM-AAC to further enhance AAC with paraphrasing augmentation and CLAP-Refine through LLMs. Our approach uses the self-supervised EAT model to extract fine-grained audio representations, which are then aligned with textual embeddings via lightweight linear layers. The caption generation LLM is efficiently fine-tuned using the LoRA adapter. Drawing inspiration from the back-translation method in machine translation, we implement paraphrasing augmentation to expand the Clotho dataset during pre-training. This strategy helps alleviate the limitation of scarce audio-text pairs and generates more diverse captions from a small set of audio clips. During inference, we introduce the plug-and-play CLAP-Refine strategy to fully exploit multiple decoding outputs, akin to the n-best rescoring strategy in speech recognition. Using the CLAP model for audio-text similarity calculation, we could select the textual descriptions generated by multiple searching beams that best match the input audio. Experimental results show that SLAM-AAC achieves state-of-the-art performance on Clotho V2 and AudioCaps, surpassing previous mainstream models.