# Symbol Emergence as an Interpersonal Multimodal Categorization

arXiv: https://arxiv.org/abs/1905.13443 · HTML: https://mlsanity.com/paper/1905.13443/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2019.

## Built on by (most cited first, 14 of 14)

- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [Emergent Communication through Metropolis-Hastings Naming Game with Deep Generative Models](https://mlsanity.com/paper/2205.12392/) (2022)
- [Generative Emergent Communication: Large Language Model is a Collective World Model](https://mlsanity.com/paper/2501.00226/) (2024)
- [Emergent Language: A Survey and Taxonomy](https://mlsanity.com/paper/2409.02645/) (2024)
- [Multiagent Multimodal Categorization for Symbol Emergence: Emergent Communication via Interpersonal Cross-modal Inference](https://mlsanity.com/paper/2109.07194/) (2021)
- [Recursive Metropolis-Hastings Naming Game: Symbol Emergence in a Multi-agent System based on Probabilistic Generative Models](https://mlsanity.com/paper/2305.19761/) (2023)
- [Metropolis-Hastings algorithm in joint-attention naming game: Experimental semiotics study](https://mlsanity.com/paper/2305.19936/) (2023)
- [Constructive Approach to Bidirectional Influence between Qualia Structure and Language Emergence](https://mlsanity.com/paper/2409.09413/) (2024)
- [Symbol Emergence as Inter-personal Categorization with Head-to-head Latent Word](https://mlsanity.com/paper/2205.15027/) (2022)
- [System 0/1/2/3: Quad-process theory for multi-timescale embodied collective cognitive systems](https://mlsanity.com/paper/2503.06138/) (2025)
- [Reward-Independent Messaging for Decentralized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.21985/) (2025)
- [SimSiam Naming Game: A Unified Approach for Emergent Communication and Representation Learning](https://mlsanity.com/paper/2410.21803/) (2024)
- [Symbol emergence as interpersonal cross-situational learning: the emergence of lexical knowledge with combinatoriality](https://mlsanity.com/paper/2306.15837/) (2023)
- [Co-Creative Learning via Metropolis-Hastings Interaction between Humans and AI](https://mlsanity.com/paper/2506.15468/) (2025)

## Abstract (arXiv)

This study focuses on category formation for individual agents and the dynamics of symbol emergence in a multi-agent system through semiotic communication. Semiotic communication is defined, in this study, as the generation and interpretation of signs associated with the categories formed through the agent's own sensory experience or by exchange of signs with other agents. From the viewpoint of language evolution and symbol emergence, organization of a symbol system in a multi-agent system is considered as a bottom-up and dynamic process, where individual agents share the meaning of signs and categorize sensory experience. A constructive computational model can explain the mutual dependency of the two processes and has mathematical support that guarantees a symbol system's emergence and sharing within the multi-agent system. In this paper, we describe a new computational model that represents symbol emergence in a two-agent system based on a probabilistic generative model for multimodal categorization. It models semiotic communication via a probabilistic rejection based on the receiver's own belief. We have found that the dynamics by which cognitively independent agents create a symbol system through their semiotic communication can be regarded as the inference process of a hidden variable in an interpersonal multimodal categorizer, if we define the rejection probability based on the Metropolis-Hastings algorithm. The validity of the proposed model and algorithm for symbol emergence is also verified in an experiment with two agents observing daily objects in the real-world environment. The experimental results demonstrate that our model reproduces the phenomena of symbol emergence, which does not require a teacher who would know a pre-existing symbol system. Instead, the multi-agent system can form and use a symbol system without having pre-existing categories.