mlsanityData to 7 Oct 2026

BATON: Aligning Text-to-Audio Model with Human Preference Feedback

arXiv 2402.00744 ↗First posted 1 Feb 2024cs.SD cs.CL eess.AScited by 18

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

With the development of AI-Generated Content (AIGC), text-to-audio models are gaining widespread attention. However, it is challenging for these models to generate audio aligned with human preference due to the inherent information density of natural language and limited model understanding ability. To alleviate this issue, we formulate the BATON, a framework designed to enhance the alignment between generated audio and text prompt using human preference feedback. Our BATON comprises three key stages: Firstly, we curated a dataset containing both prompts and the corresponding generated audio, which was then annotated based on human feedback. Secondly, we introduced a reward model using the constructed dataset, which can mimic human preference by assigning rewards to input text-audio pairs. Finally, we employed the reward model to fine-tune an off-the-shelf text-to-audio model. The experiment results demonstrate that our BATON can significantly improve the generation quality of the original text-to-audio models, concerning audio integrity, temporal relationship, and alignment with human preference.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 40 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

26 of 40 references.

References · 26

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2017Decoupled Weight Decay Regularizationcited by 1,722
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  12. 2023Training Diffusion Models with Reinforcement Learningcited by 939
14 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  4. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  5. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  6. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  7. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  8. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  9. 2022AudioGen: Textually Guided Audio Generationcited by 292
  10. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  11. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  12. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  13. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  14. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 6; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  2. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  3. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 22
  4. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 19
  5. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  6. 2024Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimizationcited by 15
  7. 2025Differentiable Reward Optimization for LLM based TTS systemcited by 15
  8. 2025MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flowscited by 14
  9. 2024AToM: Aligning Text-to-Motion Model at Event-Level with GPT-4Vision Rewardcited by 7
  10. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  11. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modelingcited by 5
  12. 2025DRAGON: Distributional Rewards Optimize Diffusion Generative Modelscited by 4
6 more
  1. 2026GSRM: Generative Speech Reward Model for Speech RLHFcited by 4
  2. 2024RiTTA: Modeling Event Relations in Text-to-Audio Generationcited by 3
  3. 2024Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learningcited by 0
  4. 2024Text Prompt is Not Enough: Sound Event Enhanced Prompt Adapter for Target Style Audio Generationcited by 0
  5. 2025Generative Multi-modal Feedback for Singing Voice Synthesis Evaluationcited by 0
  6. 2026Facial Expression Generation Aligned with Human Preference for Natural Dyadic Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Resonate: Reinforcing Text-to-Audio Generation via Online Feedback from Large Audio Language Models ↗
  2. 2025Aligning Multimodal LLM with Human Preference: A Survey
  3. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  4. 2025Aligning Audio Captions with Human Preferences
  5. 2026TuneJury: An Open Metric for Improving Music Generation Preference Alignment
  6. 2021A General Language Assistant as a Laboratory for Alignment
  7. 2026Improving Text-to-Music Generation with Human Preference Rewards
  8. 2026Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment