# The Curious Case of Representational Alignment: Unravelling Visio-Linguistic Tasks in Emergent Communication

arXiv: https://arxiv.org/abs/2407.17960 · HTML: https://mlsanity.com/paper/2407.17960/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 67 references that are in the map.
- First posted on arXiv 25 Jul 2024.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [Fast Differentiable Sorting and Ranking](https://mlsanity.com/paper/2002.08871/) (2020)
- [Multi-Agent Cooperation and the Emergence of (Natural) Language](https://mlsanity.com/paper/1612.07182/) (2016)
- [Emergent Multi-Agent Communication in the Deep Learning Era](https://mlsanity.com/paper/2006.02419/) (2020)
- [Compositionality and Generalization in Emergent Languages](https://mlsanity.com/paper/2004.09124/) (2020)
- [Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input](https://mlsanity.com/paper/1804.03984/) (2018)
- [Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog](https://mlsanity.com/paper/1706.08502/) (2017)
- [Compositional Languages Emerge in a Neural Iterated Learning Model](https://mlsanity.com/paper/2002.01365/) (2020)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [Ease-of-Teaching and Language Structure from Emergent Communication](https://mlsanity.com/paper/1906.02403/) (2019)
- [Anti-efficient encoding in emergent communication](https://mlsanity.com/paper/1905.12561/) (2019)
- [COLA: A Benchmark for Compositional Text-to-image Retrieval](https://mlsanity.com/paper/2305.03689/) (2023)
- [How agents see things: On visual representations in an emergent language game](https://mlsanity.com/paper/1808.10696/) (2018)
- [Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality](https://mlsanity.com/paper/2211.00768/) (2022)
- [EGG: a toolkit for research on Emergence of lanGuage in Games](https://mlsanity.com/paper/1907.00852/) (2019)
- [Emergent Communication of Generalizations](https://mlsanity.com/paper/2106.02668/) (2021)
- [Compositional Generalization in Unsupervised Compositional Representation Learning: A Study on Disentanglement and Emergent Language](https://mlsanity.com/paper/2210.00482/) (2022)
- [Emergent Communication: Generalization and Overfitting in Lewis Games](https://mlsanity.com/paper/2209.15342/) (2022)
- [The Emergence of Compositional Languages for Numeric Concepts Through Iterated Learning in Neural Agents](https://mlsanity.com/paper/1910.05291/) (2019)
- [Interpretable agent communication from scratch (with a generic visual processor emerging on the side)](https://mlsanity.com/paper/2106.04258/) (2021)
- [Emergent Communication for Understanding Human Language Evolution: What's Missing?](https://mlsanity.com/paper/2204.10590/) (2022)
- [Word-order biases in deep-agent emergent communication](https://mlsanity.com/paper/1905.12330/) (2019)

## Built on by (most cited first, 7 of 7)

- [Searching for Structure: Investigating Emergent Communication with Large Language Models](https://mlsanity.com/paper/2412.07646/) (2024)
- [Shaping Shared Languages: Human and Large Language Models' Inductive Biases in Emergent Communication](https://mlsanity.com/paper/2503.04395/) (2025)
- [Teacher Demonstrations in a BabyLM's Zone of Proximal Development for Contingent Multi-Turn Interaction](https://mlsanity.com/paper/2510.20411/) (2025)
- [Constructive Approach to Bidirectional Influence between Qualia Structure and Language Emergence](https://mlsanity.com/paper/2409.09413/) (2024)
- [Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect](https://mlsanity.com/paper/2507.10013/) (2025)
- [Discovering Properties of Inflectional Morphology in Neural Emergent Communication](https://mlsanity.com/paper/2508.05843/) (2025)
- [EmCom-Diffusion: Probing Visual Reflection in Emergent Languages via Image Generation](https://mlsanity.com/paper/2607.03752/) (2026)

## Abstract (arXiv)

Natural language has the universal properties of being compositional and grounded in reality. The emergence of linguistic properties is often investigated through simulations of emergent communication in referential games. However, these experiments have yielded mixed results compared to similar experiments addressing linguistic properties of human language. Here we address representational alignment as a potential contributing factor to these results. Specifically, we assess the representational alignment between agent image representations and between agent representations and input images. Doing so, we confirm that the emergent language does not appear to encode human-like conceptual visual features, since agent image representations drift away from inputs whilst inter-agent alignment increases. We moreover identify a strong relationship between inter-agent alignment and topographic similarity, a common metric for compositionality, and address its consequences. To address these issues, we introduce an alignment penalty that prevents representational drift but interestingly does not improve performance on a compositional discrimination task. Together, our findings emphasise the key role representational alignment plays in simulations of language emergence.