# Aligning Audio Captions with Human Preferences

arXiv: https://arxiv.org/abs/2509.14659 · HTML: https://mlsanity.com/paper/2509.14659/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 17 of 17)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study](https://mlsanity.com/paper/2404.10719/) (2024)
- [Reward Shaping to Mitigate Reward Hacking in RLHF](https://mlsanity.com/paper/2502.18770/) (2025)
- [Sample Efficient Reinforcement Learning with REINFORCE](https://mlsanity.com/paper/2010.11364/) (2020)
- [Can Audio Captions Be Evaluated with Image Caption Metrics?](https://mlsanity.com/paper/2110.04684/) (2021)
- [CoNeTTE: An efficient Audio Captioning system leveraging multiple datasets with Task Embedding](https://mlsanity.com/paper/2309.00454/) (2023)
- [Parameter Efficient Audio Captioning With Faithful Guidance Using Audio-text Shared Latent Representation](https://mlsanity.com/paper/2309.03340/) (2023)
- [MACE: Leveraging Audio for Evaluating Audio Captioning Systems](https://mlsanity.com/paper/2411.00321/) (2024)
- [Leveraging Audio-Only Data for Text-Queried Target Sound Extraction](https://mlsanity.com/paper/2409.13152/) (2024)
- [Human-CLAP: Human-perception-based contrastive language-audio pretraining](https://mlsanity.com/paper/2506.23553/) (2025)
- [Improving Audio Captioning Using Semantic Similarity Metrics](https://mlsanity.com/paper/2210.16470/) (2022)
- [Detecting False Alarms and Misses in Audio Captions](https://mlsanity.com/paper/2309.03326/) (2023)

## Built on by (most cited first, 1 of 1)

- [SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval](https://mlsanity.com/paper/2609.02343/) (2026)

## Abstract (arXiv)

Current audio captioning relies on supervised learning with paired audio-caption data, which is costly to curate and may not reflect human preferences in real-world scenarios. To address this, we propose a preference-aligned audio captioning framework based on Reinforcement Learning from Human Feedback (RLHF). To capture nuanced preferences, we train a Contrastive Language-Audio Pretraining (CLAP) based reward model using human-labeled pairwise preference data. This reward model is integrated into an RL framework to fine-tune any baseline captioning system without ground-truth annotations. Extensive human evaluations across multiple datasets show that our method produces captions preferred over baseline models, particularly when baselines fail to provide correct and natural captions. Furthermore, our framework achieves performance comparable to supervised approaches with ground-truth data, demonstrating effective alignment with human preferences and scalability in real-world use.