# Caregiver Talk Shapes Toddler Vision: A Computational Study of Dyadic Play

arXiv: https://arxiv.org/abs/2312.04118 · HTML: https://mlsanity.com/paper/2312.04118/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Using Shape to Categorize: Low-Shot Learning with an Explicit Shape Bias](https://mlsanity.com/paper/2101.07296/) (2021)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Self-supervised learning through the eyes of a child](https://mlsanity.com/paper/2007.16189/) (2020)
- [Towards Democratizing Joint-Embedding Self-Supervised Learning](https://mlsanity.com/paper/2303.01986/) (2023)
- [Time to augment self-supervised visual representation learning](https://mlsanity.com/paper/2207.13492/) (2022)
- [A Computational Model of Early Word Learning from the Infant's Point of View](https://mlsanity.com/paper/2006.02802/) (2020)

## Built on by (most cited first, 5 of 5)

- [Learning Object Semantic Similarity with Self-Supervision](https://mlsanity.com/paper/2405.05143/) (2024)
- [Toddlers' Active Gaze Behavior Supports Self-Supervised Object Learning](https://mlsanity.com/paper/2411.01969/) (2024)
- [Simulated Cortical Magnification Supports Self-Supervised Object Learning](https://mlsanity.com/paper/2509.15751/) (2025)
- [Re-assessing the evidence for mental rotation abilities in children using computational models](https://mlsanity.com/paper/2512.17972/) (2025)
- [Temporal Slowness in Central Vision Drives Semantic Object Learning](https://mlsanity.com/paper/2602.04462/) (2026)

## Abstract (arXiv)

Infants' ability to recognize and categorize objects develops gradually. The second year of life is marked by both the emergence of more semantic visual representations and a better understanding of word meaning. This suggests that language input may play an important role in shaping visual representations. However, even in suitable contexts for word learning like dyadic play sessions, caregivers utterances are sparse and ambiguous, often referring to objects that are different from the one to which the child attends. Here, we systematically investigate to what extent caregivers' utterances can nevertheless enhance visual representations. For this we propose a computational model of visual representation learning during dyadic play. We introduce a synthetic dataset of ego-centric images perceived by a toddler-agent that moves and rotates toy objects in different parts of its home environment while hearing caregivers' utterances, modeled as captions. We propose to model toddlers' learning as simultaneously aligning representations for 1) close-in-time images and 2) co-occurring images and utterances. We show that utterances with statistics matching those of real caregivers give rise to representations supporting improved category recognition. Our analysis reveals that a small decrease/increase in object-relevant naming frequencies can drastically impact the learned representations. This affects the attention on object names within an utterance, which is required for efficient visuo-linguistic alignment. Overall, our results support the hypothesis that caregivers' naming utterances can improve toddlers' visual representations.