# Symbol emergence as interpersonal cross-situational learning: the emergence of lexical knowledge with combinatoriality

arXiv: https://arxiv.org/abs/2306.15837 · HTML: https://mlsanity.com/paper/2306.15837/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 68 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 23 of 23)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Emergent Multi-Agent Communication in the Deep Learning Era](https://mlsanity.com/paper/2006.02419/) (2020)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Compositionality and Generalization in Emergent Languages](https://mlsanity.com/paper/2004.09124/) (2020)
- [Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input](https://mlsanity.com/paper/1804.03984/) (2018)
- [Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog](https://mlsanity.com/paper/1706.08502/) (2017)
- [Ease-of-Teaching and Language Structure from Emergent Communication](https://mlsanity.com/paper/1906.02403/) (2019)
- [Biases for Emergent Communication in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/1912.05676/) (2019)
- [Symbol Emergence in Cognitive Developmental Systems: a Survey](https://mlsanity.com/paper/1801.08829/) (2018)
- [Symbol Emergence in Robotics: A Survey](https://mlsanity.com/paper/1509.08973/) (2015)
- [Neuro-SERKET: Development of Integrative Cognitive System through the Composition of Deep Probabilistic Generative Models](https://mlsanity.com/paper/1910.08918/) (2019)
- [Emergence of Compositional Language with Deep Generational Transmission](https://mlsanity.com/paper/1904.09067/) (2019)
- [Emergent Communication through Metropolis-Hastings Naming Game with Deep Generative Models](https://mlsanity.com/paper/2205.12392/) (2022)
- [SERKET: An Architecture for Connecting Stochastic Models to Realize a Large-Scale Cognitive Model](https://mlsanity.com/paper/1712.00929/) (2017)
- [Symbol Emergence as an Interpersonal Multimodal Categorization](https://mlsanity.com/paper/1905.13443/) (2019)
- [Online Spatial Concept and Lexical Acquisition with Simultaneous Localization and Mapping](https://mlsanity.com/paper/1704.04664/) (2017)
- [Multiagent Multimodal Categorization for Symbol Emergence: Emergent Communication via Interpersonal Cross-modal Inference](https://mlsanity.com/paper/2109.07194/) (2021)
- [Spatial Concept-Based Navigation with Human Speech Instructions via Probabilistic Inference on Bayesian Generative Model](https://mlsanity.com/paper/2002.07381/) (2020)
- [Emergent Communication with World Models](https://mlsanity.com/paper/2002.09604/) (2020)
- [Symbol Emergence as Inter-personal Categorization with Head-to-head Latent Word](https://mlsanity.com/paper/2205.15027/) (2022)

## Abstract (arXiv)

We present a computational model for a symbol emergence system that enables the emergence of lexical knowledge with combinatoriality among agents through a Metropolis-Hastings naming game and cross-situational learning. Many computational models have been proposed to investigate combinatoriality in emergent communication and symbol emergence in cognitive and developmental robotics. However, existing models do not sufficiently address category formation based on sensory-motor information and semiotic communication through the exchange of word sequences within a single integrated model. Our proposed model facilitates the emergence of lexical knowledge with combinatoriality by performing category formation using multimodal sensory-motor information and enabling semiotic communication through the exchange of word sequences among agents in a unified model. Furthermore, the model enables an agent to predict sensory-motor information for unobserved situations by combining words associated with categories in each modality. We conducted two experiments with two humanoid robots in a simulated environment to evaluate our proposed model. The results demonstrated that the agents can acquire lexical knowledge with combinatoriality through interpersonal cross-situational learning based on the Metropolis-Hastings naming game and cross-situational learning. Furthermore, our results indicate that the lexical knowledge developed using our proposed model exhibits generalization performance for novel situations through interpersonal cross-modal inference.