# ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 edition

arXiv: https://arxiv.org/abs/2107.06546 · HTML: https://mlsanity.com/paper/2107.06546/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Text-Free Image-to-Speech Synthesis Using Learned Segmental Units](https://mlsanity.com/paper/2012.15454/) (2020)
- [The Zero Resource Speech Challenge 2017](https://mlsanity.com/paper/1712.04313/) (2017)
- [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019)
- [The Zero Resource Speech Challenge 2019: TTS without T](https://mlsanity.com/paper/1904.11469/) (2019)
- [Representations of language in a model of visually grounded speech signal](https://mlsanity.com/paper/1702.01991/) (2017)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [Encoding of phonology in a recurrent neural model of grounded speech](https://mlsanity.com/paper/1706.03815/) (2017)
- [Language learning using Speech to Image retrieval](https://mlsanity.com/paper/1909.03795/) (2019)
- [Semantic speech retrieval with a visually grounded model of untranscribed speech](https://mlsanity.com/paper/1710.01949/) (2017)
- [Models of Visually Grounded Speech Signal Pay Attention To Nouns: a Bilingual Experiment on English and Japanese](https://mlsanity.com/paper/1902.03052/) (2019)
- [Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech](https://mlsanity.com/paper/1909.08491/) (2019)
- [AVA-Speech: A Densely Labeled Dataset of Speech Activity in Movies](https://mlsanity.com/paper/1808.00606/) (2018)
- [Symbolic inductive bias for visually grounded learning of spoken language](https://mlsanity.com/paper/1812.09244/) (2018)
- [Textual Supervision for Visually Grounded Spoken Language Understanding](https://mlsanity.com/paper/2010.02806/) (2020)

## Built on by (most cited first, 6 of 6)

- [Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge](https://mlsanity.com/paper/2210.15759/) (2022)
- [Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling](https://mlsanity.com/paper/2202.03543/) (2022)
- [Fast-Slow Transformer for Visually Grounding Speech](https://mlsanity.com/paper/2109.08186/) (2021)
- [Evaluating context-invariance in unsupervised speech representations](https://mlsanity.com/paper/2210.15775/) (2022)
- [Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations](https://mlsanity.com/paper/2402.01298/) (2024)
- [Multimodal Input Aids a Bayesian Model of Phonetic Learning](https://mlsanity.com/paper/2407.15992/) (2024)

## Abstract (arXiv)

We present the visually-grounded language modelling track that was introduced in the Zero-Resource Speech challenge, 2021 edition, 2nd round. We motivate the new track and discuss participation rules in detail. We also present the two baseline systems that were developed for this track.