# Multiagent Multimodal Categorization for Symbol Emergence: Emergent Communication via Interpersonal Cross-modal Inference

arXiv: https://arxiv.org/abs/2109.07194 · HTML: https://mlsanity.com/paper/2109.07194/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 50 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Symbol Emergence in Cognitive Developmental Systems: a Survey](https://mlsanity.com/paper/1801.08829/) (2018)
- [Symbol Emergence in Robotics: A Survey](https://mlsanity.com/paper/1509.08973/) (2015)
- [Neuro-SERKET: Development of Integrative Cognitive System through the Composition of Deep Probabilistic Generative Models](https://mlsanity.com/paper/1910.08918/) (2019)
- [SERKET: An Architecture for Connecting Stochastic Models to Realize a Large-Scale Cognitive Model](https://mlsanity.com/paper/1712.00929/) (2017)
- [Symbol Emergence as an Interpersonal Multimodal Categorization](https://mlsanity.com/paper/1905.13443/) (2019)
- [Improved and Scalable Online Learning of Spatial Concepts and Language Models with Mapping](https://mlsanity.com/paper/1803.03481/) (2018)
- [Online Spatial Concept and Lexical Acquisition with Simultaneous Localization and Mapping](https://mlsanity.com/paper/1704.04664/) (2017)
- [Spatial Concept Acquisition for a Mobile Robot that Integrates Self-Localization and Unsupervised Word Discovery from Spoken Sentences](https://mlsanity.com/paper/1602.01208/) (2016)

## Built on by (most cited first, 10 of 10)

- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [Emergent Communication through Metropolis-Hastings Naming Game with Deep Generative Models](https://mlsanity.com/paper/2205.12392/) (2022)
- [Generative Emergent Communication: Large Language Model is a Collective World Model](https://mlsanity.com/paper/2501.00226/) (2024)
- [Emergent Language: A Survey and Taxonomy](https://mlsanity.com/paper/2409.02645/) (2024)
- [Recursive Metropolis-Hastings Naming Game: Symbol Emergence in a Multi-agent System based on Probabilistic Generative Models](https://mlsanity.com/paper/2305.19761/) (2023)
- [Metropolis-Hastings algorithm in joint-attention naming game: Experimental semiotics study](https://mlsanity.com/paper/2305.19936/) (2023)
- [Control as Probabilistic Inference as an Emergent Communication Mechanism in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.05004/) (2023)
- [Symbol Emergence as Inter-personal Categorization with Head-to-head Latent Word](https://mlsanity.com/paper/2205.15027/) (2022)
- [System 0/1/2/3: Quad-process theory for multi-timescale embodied collective cognitive systems](https://mlsanity.com/paper/2503.06138/) (2025)
- [Symbol emergence as interpersonal cross-situational learning: the emergence of lexical knowledge with combinatoriality](https://mlsanity.com/paper/2306.15837/) (2023)

## Abstract (arXiv)

This paper describes a computational model of multiagent multimodal categorization that realizes emergent communication. We clarify whether the computational model can reproduce the following functions in a symbol emergence system, comprising two agents with different sensory modalities playing a naming game. (1) Function for forming a shared lexical system that comprises perceptual categories and corresponding signs, formed by agents through individual learning and semiotic communication between agents. (2) Function to improve the categorization accuracy in an agent via semiotic communication with another agent, even when some sensory modalities of each agent are missing. (3) Function that an agent infers unobserved sensory information based on a sign sampled from another agent in the same manner as cross-modal inference. We propose an interpersonal multimodal Dirichlet mixture (Inter-MDM), which is derived by dividing an integrative probabilistic generative model, which is obtained by integrating two Dirichlet mixtures (DMs). The Markov chain Monte Carlo algorithm realizes emergent communication. The experimental results demonstrated that Inter-MDM enables agents to form multimodal categories and appropriately share signs between agents. It is shown that emergent communication improves categorization accuracy, even when some sensory modalities are missing. Inter-MDM enables an agent to predict unobserved information based on a shared sign.