# Futga: Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation

arXiv: https://arxiv.org/abs/2407.20445 · HTML: https://mlsanity.com/paper/2407.20445/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Builds on (most cited first, 26 of 26)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies](https://mlsanity.com/paper/2308.01546/) (2023)
- [AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models](https://mlsanity.com/paper/2304.00830/) (2023)
- [LP-MusicCaps: LLM-Based Pseudo Music Captioning](https://mlsanity.com/paper/2307.16372/) (2023)
- [Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning](https://mlsanity.com/paper/2308.11276/) (2023)
- [DITTO: Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2401.12179/) (2024)
- [The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation](https://mlsanity.com/paper/2311.10057/) (2023)
- [MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response](https://mlsanity.com/paper/2309.08730/) (2023)
- [MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models](https://mlsanity.com/paper/2402.06178/) (2024)
- [Video Background Music Generation: Dataset, Method and Evaluation](https://mlsanity.com/paper/2211.11248/) (2022)
- [MusCaps: Generating Captions for Music Audio](https://mlsanity.com/paper/2104.11984/) (2021)
- [MuseChat: A Conversational Music Recommendation System for Videos](https://mlsanity.com/paper/2310.06282/) (2023)
- [Toward Universal Text-to-Music Retrieval](https://mlsanity.com/paper/2211.14558/) (2022)
- [CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic Music Information Retrieval](https://mlsanity.com/paper/2304.11029/) (2023)
- [Talk the Walk: Synthetic Data Generation for Conversational Music Recommendation](https://mlsanity.com/paper/2301.11489/) (2023)
- [When the Music Stops: Tip-of-the-Tongue Retrieval for Music](https://mlsanity.com/paper/2305.14072/) (2023)
- [Music Playlist Title Generation: A Machine-Translation Approach](https://mlsanity.com/paper/2110.07354/) (2021)

## Built on by (most cited first, 5 of 5)

- [Zephyrus: An Agentic Framework for Weather Science](https://mlsanity.com/paper/2510.04017/) (2025)
- [CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation](https://mlsanity.com/paper/2410.02271/) (2024)
- [AMPS: Adaptive Modality Preference Steering via Functional Entropy](https://mlsanity.com/paper/2602.12533/) (2026)
- [SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning](https://mlsanity.com/paper/2506.15154/) (2025)
- [SongSage: A Large Musical Language Model with Lyric Generative Pre-training](https://mlsanity.com/paper/2601.01153/) (2026)

## Abstract (arXiv)

Existing music captioning methods are limited to generating concise global descriptions of short music clips, which fail to capture fine-grained musical characteristics and time-aware musical changes. To address these limitations, we propose FUTGA, a model equipped with fined-grained music understanding capabilities through learning from generative augmentation with temporal compositions. We leverage existing music caption datasets and large language models (LLMs) to synthesize fine-grained music captions with structural descriptions and time boundaries for full-length songs. Augmented by the proposed synthetic dataset, FUTGA is enabled to identify the music's temporal changes at key transition points and their musical functions, as well as generate detailed descriptions for each music segment. We further introduce a full-length music caption dataset generated by FUTGA, as the augmentation of the MusicCaps and the Song Describer datasets. We evaluate the automatically generated captions on several downstream tasks, including music generation and retrieval. The experiments demonstrate the quality of the generated captions and the better performance in various downstream tasks achieved by the proposed music captioning approach. Our code and datasets can be found in hrefhttps://huggingface.co/JoshuaW1997/FUTGAtextcolorbluehttps://huggingface.co/JoshuaW1997/FUTGA.