# HeartMuLa: A Family of Open Sourced Music Foundation Models

arXiv: https://arxiv.org/abs/2601.10547 · HTML: https://mlsanity.com/paper/2601.10547/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2026.

## Built on by (most cited first, 15 of 15)

- [SongBench: A Fine-Grained Multi-Aspect Benchmark for Song Quality Assessment](https://mlsanity.com/paper/2604.25937/) (2026)
- [Stable Audio 3](https://mlsanity.com/paper/2605.17991/) (2026)
- [LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training](https://mlsanity.com/paper/2606.30642/) (2026)
- [Pushing the Frontier of Full-Song Generation: Hierarchical Autoregressive Planning Meets Flow-Matching Rendering](https://mlsanity.com/paper/2607.20253/) (2026)
- [Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation](https://mlsanity.com/paper/2604.17656/) (2026)
- [Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance](https://mlsanity.com/paper/2607.11143/) (2026)
- [Beyond Artifacts: Towards Generalizable Synthetic Song Detection via Music-Intrinsic Features](https://mlsanity.com/paper/2606.16612/) (2026)
- [Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillation](https://mlsanity.com/paper/2606.24307/) (2026)
- [MusicMark: A Robust Generative Watermarking Framework for Music Generation](https://mlsanity.com/paper/2607.11117/) (2026)
- [VIBE: Video Instruction-aligned Background music gEneration](https://mlsanity.com/paper/2608.30125/) (2026)
- [DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation](https://mlsanity.com/paper/2609.12774/) (2026)
- [TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking](https://mlsanity.com/paper/2609.18585/) (2026)
- [Synthesis and editing of multi-instrument audio mixtures using scalar-quantised latents with MIDI Span conditioning](https://mlsanity.com/paper/2609.25546/) (2026)
- [Open-Qwen-Music: An Auditable Framework for LLM-Based Music Composition and Diffusion Rendering](https://mlsanity.com/paper/2609.31652/) (2026)
- [YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality](https://mlsanity.com/paper/2609.33757/) (2026)

## Abstract (arXiv)

We present a family of open-source Music Foundation Models designed to advance large-scale music understanding and generation across diverse tasks and modalities. Our framework consists of four major components: (1) HeartCLAP, an audio-text alignment model; (2) HeartTranscriptor, a robust lyric recognition model optimized for real-world music scenarios; and (3) HeartCodec, a low-frame-rate (12.5 Hz) yet high-fidelity music codec tokenizer that captures long-range musical structure while preserving fine-grained acoustic details and enabling efficient autoregressive modeling; (4) HeartMuLa, an LLM-based song generation model capable of synthesizing high-fidelity music under rich, user-controllable conditions (e.g., textual style descriptions, lyrics, and reference audio). In addition, it provides two specialized modes: (i) fine-grained musical attribute control, which allows users to specify the style of different song sections (e.g., intro, verse, chorus) using natural language prompts; and (ii) short, engaging music generation, which is suitable as background music for short videos. Lastly, HeartMuLa improves significantly when scaled to 7B parameters. For the first time, we show that a Suno-level, commercial-grade system can be reproduced using academic-scale data and GPU resources. We expect these foundation models to serve as strong baselines for future research and to facilitate practical applications in multimodal content production.