# SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set

arXiv: https://arxiv.org/abs/1707.08435 · HTML: https://mlsanity.com/paper/1707.08435/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2017.

## Built on by (most cited first, 8 of 8)

- [Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data](https://mlsanity.com/paper/2410.18558/) (2024)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigation](https://mlsanity.com/paper/2109.14200/) (2021)
- [Evaluation of Audio-Visual Alignments in Visually Grounded Speech Models](https://mlsanity.com/paper/2108.02562/) (2021)
- [Vision-Speech Models: Teaching Speech Models to Converse about Images](https://mlsanity.com/paper/2503.15633/) (2025)
- [Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset](https://mlsanity.com/paper/2110.07575/) (2021)
- [NativQA Framework: Enabling LLMs and VLMs with Native, Local, and Everyday Knowledge](https://mlsanity.com/paper/2504.05995/) (2025)
- [Vision Transformers Learn Word Meaning from Images Alone](https://mlsanity.com/paper/2610.13252/) (2026)

## Abstract (arXiv)

This paper presents an augmentation of MSCOCO dataset where speech is added to image and text. Speech captions are generated using text-to-speech (TTS) synthesis resulting in 616,767 spoken captions (more than 600h) paired with images. Disfluencies and speed perturbation are added to the signal in order to sound more natural. Each speech signal (WAV) is paired with a JSON file containing exact timecode for each word/syllable/phoneme in the spoken caption. Such a corpus could be used for Language and Vision (LaVi) tasks including speech input or output instead of text. Investigating multimodal learning schemes for unsupervised speech pattern discovery is also possible with this corpus, as demonstrated by a preliminary study conducted on a subset of the corpus (10h, 10k spoken captions). The dataset is available on Zenodo: https://zenodo.org/record/4282267