# Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproduction

arXiv: https://arxiv.org/abs/2402.03618 · HTML: https://mlsanity.com/paper/2402.03618/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Builds on (most cited first, 10 of 10)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks](https://mlsanity.com/paper/2311.09247/) (2023)
- [Gibbs Sampling with People](https://mlsanity.com/paper/2008.02595/) (2020)
- [Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines](https://mlsanity.com/paper/2205.11558/) (2022)
- [SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction](https://mlsanity.com/paper/2312.03035/) (2023)
- [Learning to solve complex tasks by growing knowledge culturally across generations](https://mlsanity.com/paper/2107.13377/) (2021)

## Built on by (most cited first, 2 of 2)

- [Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective](https://mlsanity.com/paper/2412.12276/) (2024)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)

## Abstract (arXiv)

Humans extract useful abstractions of the world from noisy sensory data. Serial reproduction allows us to study how people construe the world through a paradigm similar to the game of telephone, where one person observes a stimulus and reproduces it for the next to form a chain of reproductions. Past serial reproduction experiments typically employ a single sensory modality, but humans often communicate abstractions of the world to each other through language. To investigate the effect language on the formation of abstractions, we implement a novel multimodal serial reproduction framework by asking people who receive a visual stimulus to reproduce it in a linguistic format, and vice versa. We ran unimodal and multimodal chains with both humans and GPT-4 and find that adding language as a modality has a larger effect on human reproductions than GPT-4's. This suggests human visual and linguistic representations are more dissociable than those of GPT-4.