IteraTTA: An interface for exploring both text prompts and audio priors in generating music with text-to-audio models
Not on the map. Why?
Abstract (arXiv)
Recent text-to-audio generation techniques have the potential to allow novice users to freely generate music audio. Even if they do not have musical knowledge, such as about chord progressions and instruments, users can try various text prompts to generate audio. However, compared to the image domain, gaining a clear understanding of the space of possible music audios is difficult because users cannot listen to the variations of the generated audios simultaneously. We therefore facilitate users in exploring not only text prompts but also audio priors that constrain the text-to-audio music generation process. This dual-sided exploration enables users to discern the impact of different text prompts and audio priors on the generation results through iterative comparison of them. Our developed interface, IteraTTA, is specifically designed to aid users in refining text prompts and selecting favorable audio priors from the generated audios. With this, users can progressively reach their loosely-specified goals while understanding and exploring the space of possible results. Our implementation and discussions highlight design considerations that are specifically required for text-to-audio models and how interaction techniques can contribute to their effectiveness.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 33 of its 64 references that are in the map. First posted on arXiv 24 Jul 2023.
Builds on
33 of 64 references.
References · 33
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
- 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
- 2020Jukebox: A Generative Model for Musiccited by 535
- 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
- 2023MusicLM: Generating Music From Textcited by 442
- 2019Clotho: An Audio Captioning Datasetcited by 354
- 2018Music Transformercited by 207
- 2021Symbolic Music Generation with Diffusion Modelscited by 152
- 2021Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphscited by 109
21 more
- 2017MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generationcited by 72
- 2021RAVE: A variational autoencoder for fast and high-quality neural audio synthesiscited by 72
- 2020A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directionscited by 56
- 2022A Taxonomy of Prompt Modifiers for Text-To-Image Generationcited by 54
- 2021EMOPIA: A Multi-Modal Pop Piano Dataset For Emotion Recognition and Emotion-based Music Generationcited by 53
- 2022Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generationcited by 48
- 2020Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modellingcited by 45
- 2020Learning Interpretable Representation for Controllable Polyphonic Music Generationcited by 40
- 2023Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineeringcited by 40
- 2021Controllable deep melody generation via hierarchical music structure representationcited by 30
- 2020Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythmcited by 28
- 2020AI Song Contest: Human-AI Co-Creation in Songwritingcited by 25
- 2017Music Generation by Deep Learning - Challenges and Directionscited by 24
- 2022Musika! Fast Infinite Waveform Music Generationcited by 24
- 2021Creativity and Machine Learning: A Surveycited by 21
- 2018Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generationcited by 20
- 2014AI Methods in Algorithmic Composition: A Comprehensive Surveycited by 12
- 2022Generating music with sentiment using Transformer-GANscited by 11
- 2020Generative Melody Composition with Human-in-the-Loop Bayesian Optimizationcited by 6
- 2021A Benchmarking Initiative for Audio-Domain Music Generation Using the Freesound Loop Datasetcited by 4
- 2021Tool- and Domain-Agnostic Parameterization of Style Transfer Effects Leveraging Pretrained Perceptual Metricscited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.