mlsanityData to 7 Oct 2026

Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speech

arXiv 2410.14101 ↗First posted 18 Oct 2024cs.SD cs.AI eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Visual Text-to-Speech (VTTS) aims to take the environmental image as the prompt to synthesize reverberant speech for the spoken content. Previous works focus on the RGB modality for global environmental modeling, overlooking the potential of multi-source spatial knowledge like depth, speaker position, and environmental semantics. To address these issues, we propose a novel multi-source spatial knowledge understanding scheme for immersive VTTS, termed MS2KU-VTTS. Specifically, we first prioritize RGB image as the dominant source and consider depth image, speaker position knowledge from object detection, and Gemini-generated semantic captions as supplementary sources. Afterwards, we propose a serial interaction mechanism to effectively integrate both dominant and supplementary sources. The resulting multi-source knowledge is dynamically integrated based on the respective contributions of each source.This enriched interaction and integration of multi-source spatial knowledge guides the speech generation model, enhancing the immersive speech experience. Experimental results demonstrate that the MS2KU-VTTS surpasses existing baselines in generating immersive speech. Demos and code are available at: https://github.com/AI-S2-Lab/MS2KU-VTTS.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

33 of 40 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  8. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  9. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  10. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  11. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  12. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
21 more
  1. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speechcited by 147
  2. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  3. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 74
  4. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  5. 2022Visual Acoustic Matchingcited by 45
  6. 2023Novel-View Acoustic Synthesiscited by 33
  7. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  8. 2021Learning Audio-Visual Dereverberationcited by 26
  9. 2023ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformercited by 24
  10. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  11. 2023Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modelingcited by 17
  12. 2023Self-Supervised Visual Acoustic Matchingcited by 14
  13. 2023LAVSS: Location-Guided Audio-Visual Spatial Audio Separationcited by 8
  14. 2024MCDubber: Multimodal Context-Aware Expressive Video Dubbingcited by 7
  15. 2022Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generationcited by 6
  16. 2024Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modelingcited by 6
  17. 2021Environment Aware Text-to-Speech Synthesiscited by 4
  18. 2024Speak in the Scene: Diffusion-based Acoustic Scene Transfer toward Immersive Speech Generationcited by 4
  19. 2024DIFFRENT: A Diffusion Model for Recording Environment Transfer of Speechcited by 3
  20. 2018RLS-Based Adaptive Dereverberation Tracing Abrupt Position Change of Target Speakercited by 1
  21. 2021Directional MCLP Analysis and Reconstruction for Spatial Speech Communicationcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 4.

4 papers.

  1. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  2. 2024Intra- and Inter-modal Context Interaction Modeling for Conversational Speech Synthesiscited by 5
  3. 2024Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speechcited by 4
  4. 2024Towards Expressive Video Dubbing with Multiscale Multimodal Context Interactioncited by 2