mlsanityData to 7 Oct 2026

Seeing What You Say: Expressive Image Generation from Speech

arXiv 2511.03423 ↗First posted 5 Nov 2025eess.AS cs.CV cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper proposes VoxStudio, the first unified and end-to-end speech-to-image model that generates expressive images directly from spoken descriptions by jointly aligning linguistic and paralinguistic information. At its core is a speech information bottleneck (SIB) module, which compresses raw speech into compact semantic tokens, preserving prosody and emotional nuance. By operating directly on these tokens, VoxStudio eliminates the need for an additional speech-to-text system, which often ignores the hidden details beyond text, e.g., tone or emotion. We also release VoxEmoset, a large-scale paired emotional speech-image dataset built via an advanced TTS engine to affordably generate richly expressive utterances. Comprehensive experiments on the SpokenCOCO, Flickr8kAudio, and VoxEmoset benchmarks demonstrate the feasibility of our method and highlight key challenges, including emotional consistency and linguistic ambiguity, paving the way for future research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 65 references that are in the map. First posted on arXiv 5 Nov 2025.

Builds on

43 of 65 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  12. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
31 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  2. 2024Qwen2-Audio Technical Reportcited by 543
  3. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  4. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  5. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  6. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  7. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 165
  8. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  9. 2020Seen and Unseen emotional style transfer for voice conversion with a new emotional speech datasetcited by 119
  10. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  11. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  12. 2023EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributescited by 81
  13. 2018Seeing Voices and Hearing Faces: Cross-modal biometric matchingcited by 60
  14. 2018Perfect match: Improved cross-modal embeddings for audio-visual synchronisationcited by 60
  15. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  16. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  17. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  18. 2021Audio Transformerscited by 41
  19. 2023The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusioncited by 41
  20. 2024EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmarkcited by 39
  21. 2021Language-Driven Image Style Transfercited by 36
  22. 2024EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Modelscited by 35
  23. 2023Imaginary Voice: Face-styled Diffusion Model for Text-to-Speechcited by 27
  24. 2024EmoEdit: Evoking Emotions through Image Manipulationcited by 27
  25. 2022TVLT: Textless Vision-Language Transformercited by 22
  26. 2021Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Associationcited by 21
  27. 2024EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal Promptscited by 14
  28. 2020S2IGAN: Speech-to-Image Generation via Adversarial Learningcited by 8
  29. 2024TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languagescited by 4
  30. 2024Whisper-SV: Adapting Whisper for Low-data-resource Speaker Verificationcited by 4
  31. 2024SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image datacited by 2

Built on by

Not cited yet.