# S2IGAN: Speech-to-Image Generation via Adversarial Learning

arXiv: https://arxiv.org/abs/2005.06968 · HTML: https://mlsanity.com/paper/2005.06968/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2020.

## Built on by (most cited first, 8 of 8)

- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation](https://mlsanity.com/paper/2107.00249/) (2021)
- [AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style](https://mlsanity.com/paper/2103.13722/) (2021)
- [Speech-to-SQL: Towards Speech-driven SQL Query Generation From Natural Language Question](https://mlsanity.com/paper/2201.01209/) (2022)
- [TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages](https://mlsanity.com/paper/2402.16021/) (2024)
- [SViQA: A Unified Speech-Vision Multimodal Model for Textless Visual Question Answering](https://mlsanity.com/paper/2504.01049/) (2025)
- [Seeing What You Say: Expressive Image Generation from Speech](https://mlsanity.com/paper/2511.03423/) (2025)
- [Speak the Art: A Direct Speech to Image Generation Framework](https://mlsanity.com/paper/2601.00827/) (2025)

## Abstract (arXiv)

An estimated half of the world's languages do not have a written form, making it impossible for these languages to benefit from any existing text-based technologies. In this paper, a speech-to-image generation (S2IG) framework is proposed which translates speech descriptions to photo-realistic images without using any text information, thus allowing unwritten languages to potentially benefit from this technology. The proposed S2IG framework, named S2IGAN, consists of a speech embedding network (SEN) and a relation-supervised densely-stacked generative model (RDG). SEN learns the speech embedding with the supervision of the corresponding visual information. Conditioned on the speech embedding produced by SEN, the proposed RDG synthesizes images that are semantically consistent with the corresponding speech descriptions. Extensive experiments on two public benchmark datasets CUB and Oxford-102 demonstrate the effectiveness of the proposed S2IGAN on synthesizing high-quality and semantically-consistent images from the speech signal, yielding a good performance and a solid baseline for the S2IG task.