# Speak in the Scene: Diffusion-based Acoustic Scene Transfer toward Immersive Speech Generation

arXiv: https://arxiv.org/abs/2406.12688 · HTML: https://mlsanity.com/paper/2406.12688/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2024.

## Built on by (most cited first, 4 of 4)

- [Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2412.06602/) (2024)
- [Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speech](https://mlsanity.com/paper/2410.14101/) (2024)
- [UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matching](https://mlsanity.com/paper/2506.09874/) (2025)
- [VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis](https://mlsanity.com/paper/2602.02591/) (2026)

## Abstract (arXiv)

This paper introduces a novel task in generative speech processing, Acoustic Scene Transfer (AST), which aims to transfer acoustic scenes of speech signals to diverse environments. AST promises an immersive experience in speech perception by adapting the acoustic scene behind speech signals to desired environments. We propose AST-LDM for the AST task, which generates speech signals accompanied by the target acoustic scene of the reference prompt. Specifically, AST-LDM is a latent diffusion model conditioned by CLAP embeddings that describe target acoustic scenes in either audio or text modalities. The contributions of this paper include introducing the AST task and implementing its baseline model. For AST-LDM, we emphasize its core framework, which is to preserve the input speech and generate audio consistently with both the given speech and the target acoustic environment. Experiments, including objective and subjective tests, validate the feasibility and efficacy of our approach.