# Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation

arXiv: https://arxiv.org/abs/2406.10970 · HTML: https://mlsanity.com/paper/2406.10970/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2024.

## Built on by (most cited first, 28 of 28)

- [Presto! Distilling Steps and Layers for Accelerating Music Generation](https://mlsanity.com/paper/2410.05167/) (2024)
- [Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer](https://mlsanity.com/paper/2410.05151/) (2024)
- [MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners](https://mlsanity.com/paper/2506.18729/) (2025)
- [MusGO: A Community-Driven Framework For Assessing Openness in Music-Generative AI](https://mlsanity.com/paper/2507.03599/) (2025)
- [Stable Audio 3](https://mlsanity.com/paper/2605.17991/) (2026)
- [MR-FlowDPO: Multi-Reward Direct Preference Optimization for Flow-Matching Text-to-Music Generation](https://mlsanity.com/paper/2512.10264/) (2025)
- [Music for All: Representational Bias and Cross-Cultural Adaptability of Music Generation Models](https://mlsanity.com/paper/2502.07328/) (2025)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [SyMuPe: Affective and Controllable Symbolic Music Performance](https://mlsanity.com/paper/2511.03425/) (2025)
- [Subtractive Training for Music Stem Insertion using Latent Diffusion Models](https://mlsanity.com/paper/2406.19328/) (2024)
- [Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation](https://mlsanity.com/paper/2506.08570/) (2025)
- [Low-Resource Guidance for Controllable Latent Audio Diffusion](https://mlsanity.com/paper/2603.04366/) (2026)
- [The Interpretation Gap in Text-to-Music Generation Models](https://mlsanity.com/paper/2407.10328/) (2024)
- [Genre Controlled Music Generation via Activation Steering](https://mlsanity.com/paper/2506.10225/) (2025)
- [AI-Assisted Music Production: A User Study on Text-to-Music Models](https://mlsanity.com/paper/2509.23364/) (2025)
- [Stemphonic: All-at-once Flexible Multi-stem Music Generation](https://mlsanity.com/paper/2602.09891/) (2026)
- [Evaluating Disentangled Representations for Controllable Music Generation](https://mlsanity.com/paper/2602.10058/) (2026)
- [Latent Space Disentanglement via Activation Steering for Interpretable Attribute Control in Symbolic Music Generation](https://mlsanity.com/paper/2605.31295/) (2026)
- [StepAudio 3 Music Technical Report](https://mlsanity.com/paper/2609.16034/) (2026)
- [Exploring Adapter Design Tradeoffs for Low Resource Music Generation](https://mlsanity.com/paper/2506.21298/) (2025)
- [MindMelody: A Closed-Loop EEG-Driven System for Personalized Music Intervention](https://mlsanity.com/paper/2605.01235/) (2026)
- [Auditing Training Data in Generative Music Models via Black-Box Membership Inference](https://mlsanity.com/paper/2605.29202/) (2026)
- [SketchSong: Hierarchical Song Generation with Sketch Planning and Fine-Grained Multi-Track Modeling](https://mlsanity.com/paper/2606.03169/) (2026)
- [InvFlowFD: Reference-Free and Background-Set-Free Perceptual Music Quality Metric with Flow Matching Inversion](https://mlsanity.com/paper/2608.04142/) (2026)
- [MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation](https://mlsanity.com/paper/2608.09035/) (2026)
- [DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation](https://mlsanity.com/paper/2609.12774/) (2026)
- [YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality](https://mlsanity.com/paper/2609.33757/) (2026)
- [TrueMuse: A Benchmark for Data Attribution in Text-to-Music Models](https://mlsanity.com/paper/2610.00835/) (2026)

## Abstract (arXiv)

We present JASCO, a temporally controlled text-to-music generation model utilizing both symbolic and audio-based conditions. JASCO can generate high-quality music samples conditioned on global text descriptions along with fine-grained local controls. JASCO is based on the Flow Matching modeling paradigm together with a novel conditioning method. This allows music generation controlled both locally (e.g., chords) and globally (text description). Specifically, we apply information bottleneck layers in conjunction with temporal blurring to extract relevant information with respect to specific controls. This allows the incorporation of both symbolic and audio-based conditions in the same text-to-music model. We experiment with various symbolic control signals (e.g., chords, melody), as well as with audio representations (e.g., separated drum tracks, full-mix). We evaluate JASCO considering both generation quality and condition adherence, using both objective metrics and human studies. Results suggest that JASCO is comparable to the evaluated baselines considering generation quality while allowing significantly better and more versatile controls over the generated music. Samples are available on our demo page https://pages.cs.huji.ac.il/adiyoss-lab/JASCO.