mlsanityData to 7 Oct 2026

Text-Free Image-to-Speech Synthesis Using Learned Segmental Units

arXiv 2012.15454 ↗First posted 31 Dec 2020cs.CL cs.AI cs.CVcited by 50

Not on the map. Why?

Abstract (arXiv)

In this paper we present the first model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supervision. Instead, we connect the image captioning module and the speech synthesis module with a set of discrete, sub-word speech units that are discovered with a self-supervised visual grounding task. We conduct experiments on the Flickr8k spoken caption dataset in addition to a novel corpus of spoken audio captions collected for the popular MSCOCO dataset, demonstrating that our generated captions also capture diverse visual semantics of the images they describe. We investigate several different intermediate speech representations, and empirically find that the representation must satisfy several important properties to serve as drop-in replacements for text.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 9; 2022: 11; 2023: 12; 2024: 14; 2025: 4.

50 papers.

Sort:

By area:

  1. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  2. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  3. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  4. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  5. 2022Audio Self-supervised Learning: A Surveycited by 65
  6. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  7. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  8. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  9. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  10. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  11. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  12. 2022WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Modelscited by 25
38 more
  1. 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
  2. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  3. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  4. 2023Wave to Syntax: Probing spoken language models for syntaxcited by 18
  5. 2022Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognitioncited by 14
  6. 2021VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Overcited by 13
  7. 2021Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigationcited by 12
  8. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  9. 2022M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrievalcited by 9
  10. 2022ConceptBeam: Concept Driven Target Speech Extractioncited by 8
  11. 2021ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 editioncited by 6
  12. 2021Semantic sentence similarity: size does not always mattercited by 5
  13. 2022Video-Guided Curriculum Learning for Spoken Video Groundingcited by 5
  14. 2023Visually grounded few-shot word learning in low-resource settingscited by 4
  15. 2023Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokenscited by 4
  16. 2024TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languagescited by 4
  17. 2024RU-AI: A Large Multimodal Dataset for Machine-Generated Content Detectioncited by 4
  18. 2024You Only Speak Once to Seecited by 4
  19. 2025Vision-Speech Models: Teaching Speech Models to Converse about Imagescited by 4
  20. 2024A model of early word acquisition based on realistic-scale audiovisual naming eventscited by 3
  21. 2024Measuring Sound Symbolism in Audio-visual Modelscited by 3
  22. 2021Spoken ObjectNet: A Bias-Controlled Spoken Caption Datasetcited by 2
  23. 2023Visually Grounded Keyword Detection and Localisation for Low-Resource Languagescited by 2
  24. 2023Cascading and Direct Approaches to Unsupervised Constituency Parsing on Spoken Sentencescited by 2
  25. 2023Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning Systemcited by 2
  26. 2023Audio-Visual Neural Syntax Acquisitioncited by 2
  27. 2024VHASR: A Multimodal Speech Recognition System With Vision Hotwordscited by 2
  28. 2025Towards Unsupervised Speech Recognition at the Syllable-Levelcited by 2
  29. 2022Is my automatic audio captioning system so bad? spider-max: a metric to consider several caption candidatescited by 1
  30. 2023Visually grounded few-shot word acquisition with fewer shotscited by 1
  31. 2024Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Modelcited by 1
  32. 2024Coarse-to-fine Alignment Makes Better Speech-image Retrievalcited by 1
  33. 2025The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoderscited by 1
  34. 2024Braille-to-Speech Generator: Audio Generation Based on Joint Fine-Tuning of CLIP and Fastspeech2cited by 0
  35. 2024Survey: Transformer-based Models in Data Modality Conversioncited by 0
  36. 2024Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrievalcited by 0
  37. 2024Grammar Induction from Visual, Speech and Textcited by 0
  38. 2025Seeing What You Say: Expressive Image Generation from Speechcited by 0