mlsanityData to 14 Oct 2026

S2Accompanist: A Semantic-Aware and Structure-Guided Diffusion Model for Music Accompaniment Generation

arXiv 2605.17414 ↗First posted 17 May 2026eess.AScited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

High-fidelity text-to-music generation typically relies on massive proprietary datasets and immense computational resources. Existing models often struggle to generate coherent pure musical accompaniments and lack precise, localized semantic control due to their reliance on coarse, track-level annotations. To address these limitations under constrained data and computing resources, we propose S2Accompanist, a Semantic-Aware and Structure-Guided Diffusion Model developed for the ICME2026 ATTM Grand Challenge. Specifically, we design an automated data pipeline comprising structural segmentation, Large Audio-Language Model driven segment-level captioning, and dual-metric quality grading to overcome the absence of localized metadata in raw datasets. Furthermore, we propose a semantic-aware Variational Autoencoder fine-tuning strategy that explicitly distills foundational LeadSheet structures into the acoustic latent space, effectively improving the overall audio fidelity. Extensive experiments demonstrate that S2Accompanist achieves state-of-the-art objective performance on the ATTM Grand Challenge benchmark across both the Efficiency and Performance Tracks. With only 402M parameters, our model remains competitive compared to larger-scale unconstrained models and secured first place in the Efficiency Track.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 20 of its 26 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

20 of 26 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching27 Oct
  2. 2025ACE-Step: A Step Towards Music Generation Foundation Model28 May
  3. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion3 Mar
  4. 2025InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generation28 Feb
  5. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies3 Aug
  6. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Models8 Feb

Other references · 14

  1. 2025Qwen3-Omni Technical Reportcited by 497
  2. 2023Simple and Controllable Music Generationcited by 488
  3. 2023MusicLM: Generating Music From Textcited by 448
  4. 2024Stable Audio Opencited by 197
  5. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  6. 2025YuE: Scaling Open Foundation Models for Long-Form Music Generationcited by 71
  7. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 68
  8. 2019Demucs: Deep Extractor for Music Sources with extra unlabeled data remixedcited by 54
  9. 2026ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generationcited by 26
  10. 2022Melody transcription via generative pre-trainingcited by 24
  11. 2025SongFormer: Scaling Music Structure Analysis with Heterogeneous Supervisioncited by 16
  12. 2024MusicFlow: Cascaded Flow Matching for Text Guided Music Generationcited by 15
2 more
  1. 2025DiffRhythm+: Controllable and Flexible Full-Length Song Generation with Preference Optimizationcited by 12
  2. 2025Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesiscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models
  2. 2024Quality-aware Masked Diffusion Transformer for Enhanced Music Generation
  3. 2026Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes
  4. 2026LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training
  5. 2024Improving Musical Accompaniment Co-creation via Diffusion Transformers
  6. 2025Generation of Musical Timbres using a Text-Guided Diffusion Model
  7. 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
  8. 2023Efficient Neural Music Generation