# Learning English with Peppa Pig

arXiv: https://arxiv.org/abs/2202.12917 · HTML: https://mlsanity.com/paper/2202.12917/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 25 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [AVLnet: Learning Audio-Visual Language Representations from Instructional Videos](https://mlsanity.com/paper/2006.09199/) (2020)
- [Deep Fragment Embeddings for Bidirectional Image Sentence Mapping](https://mlsanity.com/paper/1406.5679/) (2014)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions](https://mlsanity.com/paper/2105.04489/) (2021)
- [Representations of language in a model of visually grounded speech signal](https://mlsanity.com/paper/1702.01991/) (2017)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [Learning Word-Like Units from Joint Audio-Visual Analysis](https://mlsanity.com/paper/1701.07481/) (2017)
- [Encoding of phonology in a recurrent neural model of grounded speech](https://mlsanity.com/paper/1706.03815/) (2017)
- [Fast-Slow Transformer for Visually Grounding Speech](https://mlsanity.com/paper/2109.08186/) (2021)
- [Language learning using Speech to Image retrieval](https://mlsanity.com/paper/1909.03795/) (2019)
- [Transfer Learning from Audio-Visual Grounding to Speech Recognition](https://mlsanity.com/paper/1907.04355/) (2019)
- [Towards Visually Grounded Sub-Word Speech Unit Discovery](https://mlsanity.com/paper/1902.08213/) (2019)
- [Models of Visually Grounded Speech Signal Pay Attention To Nouns: a Bilingual Experiment on English and Japanese](https://mlsanity.com/paper/1902.03052/) (2019)
- [Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech](https://mlsanity.com/paper/1909.08491/) (2019)
- [Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigation](https://mlsanity.com/paper/2109.14200/) (2021)
- [Narration Generation for Cartoon Videos](https://mlsanity.com/paper/2101.06803/) (2021)

## Built on by (most cited first, 8 of 8)

- [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022)
- [BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models](https://mlsanity.com/paper/2306.01506/) (2023)
- [Putting Natural in Natural Language Processing](https://mlsanity.com/paper/2305.04572/) (2023)
- [Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model](https://mlsanity.com/paper/2305.11435/) (2023)
- [ConceptBeam: Concept Driven Target Speech Extraction](https://mlsanity.com/paper/2207.11964/) (2022)
- [A model of early word acquisition based on realistic-scale audiovisual naming events](https://mlsanity.com/paper/2406.05259/) (2024)
- [Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priors](https://mlsanity.com/paper/2603.08359/) (2026)
- [Analysing the Impact of Audio Quality on the Use of Naturalistic Long-Form Recordings for Infant-Directed Speech Research](https://mlsanity.com/paper/2305.01965/) (2023)

## Abstract (arXiv)

Recent computational models of the acquisition of spoken language via grounding in perception exploit associations between the spoken and visual modalities and learn to represent speech and visual data in a joint vector space. A major unresolved issue from the point of ecological validity is the training data, typically consisting of images or videos paired with spoken descriptions of what is depicted. Such a setup guarantees an unrealistically strong correlation between speech and the visual data. In the real world the coupling between the linguistic and the visual modality is loose, and often confounded by correlations with non-semantic aspects of the speech signal. Here we address this shortcoming by using a dataset based on the children's cartoon Peppa Pig. We train a simple bi-modal architecture on the portion of the data consisting of dialog between characters, and evaluate on segments containing descriptive narrations. Despite the weak and confounded signal in this training data our model succeeds at learning aspects of the visual semantics of spoken language.