mlsanityData to 14 Oct 2026

BackgroundMellow: A Multi-Modal Cohesive Framework for Narrative-Driven Rich Cinematic Soundscape Generation

arXiv 2607.11364 ↗First posted 13 Jul 2026cs.LG cs.AI cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

Generating immersive, synchronized and cinematic audio for long-form textual narratives remains a significant challenge in multi-modal AI. While current Text-to-Audio (TTA) frameworks successfully synthesize isolated sound effects, they struggle with narrative cohesion, temporal alignment, and cinematic emotional depth. We present BackgroundMellow, a framework that treats story-to-audio generation as a precise orchestration and signal processing problem. This framework is enabled without ground-truth through a master-specialist agent architecture that decomposes text into precise and multi-layered audio cues, generates each category of sounds with suitable specialist model, and superimposes the soundscapes to create a unified and aligned audio segment. Our pipeline is built over Tango2 latent diffusion model for environmental synthesis alongside a novel Cinematic BGM Retriever mined from professional soundtracks. To automate the sound mixing process, we use an NLP based module that predicts precise audio parameters, like start time, duration, and relative loudness, based on the narrative timeline. We further empirically evaluate and show the efficacy of the proposed framework leveraging nearest-neighbor retrieval against a curated dataset of YouTube cinematic trailers to measure temporal synchronization, coverage, and spectral richness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 25 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

15 of 25 references.

References · 15

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  4. 2023Simple and Controllable Music Generationcited by 488
  5. 2023MusicLM: Generating Music From Textcited by 448
  6. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  7. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  8. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  9. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  10. 2025Aligning Text-to-Music Evaluation with Human Preferencescited by 21
  11. 2018Bach2Bach: Generating Music Using A Deep Reinforcement Learning Approachcited by 4
  12. 2025AudioStory: Generating Long-Form Narrative Audio with Large Language Modelscited by 3
3 more
  1. 2024M2M-Gen: A Multimodal Framework for Automated Background Music Generation in Japanese Manga Using Large Language Modelscited by 2
  2. 2024MetaBGM: Dynamic Soundtrack Transformation For Continuous Multi-Scene Experiences With Ambient Awareness And Personalizationcited by 1
  3. 2025A Multi-Agent AI Framework for Immersive Audiobook Production through Spatial Audio and Neural Narrationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions
  2. 2026Semantic visually-guided acoustic highlighting with large vision-language models
  3. 2024SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation
  4. 2026SemanticAudio: Audio Generation and Editing in Semantic Space
  5. 2026Soundwich: Video Generation with Layered and Controllable Audio
  6. 2026Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text
  7. 2026Audible World Models: Spatially Aware Sound Generation for 3D Worlds
  8. 2025EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation