mlsanityData to 7 Oct 2026

RiTTA: Modeling Event Relations in Text-to-Audio Generation

arXiv 2412.15922 ↗First posted 20 Dec 2024cs.LG cs.SD eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

Despite significant advancements in Text-to-Audio (TTA) generation models achieving high-fidelity audio with fine-grained context understanding, they struggle to model the relations between audio events described in the input text. However, previous TTA methods have not systematically explored audio event relation modeling, nor have they proposed frameworks to enhance this capability. In this work, we systematically study audio event relation modeling in TTA generation models. We first establish a benchmark for this task by: 1. proposing a comprehensive relation corpus covering all potential relations in real-world scenarios; 2. introducing a new audio event corpus encompassing commonly heard audios; and 3. proposing new evaluation metrics to assess audio event relation modeling from various perspectives. Furthermore, we propose a finetuning framework to enhance existing TTA models ability to model audio events relation. Code is available at: https://github.com/yuhanghe01/RiTTA

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

43 of 61 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  10. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  11. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  12. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
31 more
  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  2. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  3. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  4. 2023Simple and Controllable Music Generationcited by 480
  5. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  6. 2023MusicLM: Generating Music From Textcited by 442
  7. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  8. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  9. 2022AudioGen: Textually Guided Audio Generationcited by 292
  10. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  11. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  12. 2019Sparse Sequence-to-Sequence Modelscited by 167
  13. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  14. 2021Taming Visually Guided Sound Generationcited by 136
  15. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  16. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  17. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  18. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  19. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  20. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
  21. 2020Generating Visually Aligned Sound from Videoscited by 67
  22. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  23. 2023WavJourney: Compositional Audio Creation with Large Language Modelscited by 27
  24. 2023Audio-Text Models Do Not Yet Leverage Natural Languagecited by 26
  25. 2024AudioTime: A Temporally-aligned Audio-text Benchmark Datasetcited by 19
  26. 2024BATON: Aligning Text-to-Audio Model with Human Preference Feedbackcited by 18
  27. 2023Acoustic Prompt Tuning: Empowering Large Language Models with Audition Capabilitiescited by 10
  28. 2021SoundDet: Polyphonic Moving Sound Event Detection and Localization from Raw Waveformcited by 7
  29. 2022Hyperbolic Audio Source Separationcited by 7
  30. 2024Prompt Tuning for Audio Deepfake Detection: Computationally Efficient Test-time Domain Adaptation with Limited Target Datasetcited by 7
  31. 2018A Study of the Complexity and Accuracy of Direction of Arrival Estimation Methods Based on GCC-PHAT for a Pair of Close Microphonescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Modelscited by 0
  2. 2026TORUS: A Test of Rendering-Understanding Self-Coherence for Unified Audio Modelscited by 0
  3. 2026AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluationcited by 0