# Emergent communication for AR

arXiv: https://arxiv.org/abs/2308.07342 · HTML: https://mlsanity.com/paper/2308.07342/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 14 references that are in the map.
- First posted on arXiv 12 Aug 2023.

## Builds on (most cited first, 7 of 7)

- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Emergent Communication of Generalizations](https://mlsanity.com/paper/2106.02668/) (2021)
- [Semantic Importance-Aware Communications Using Pre-trained Language Models](https://mlsanity.com/paper/2302.07142/) (2023)
- [Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication](https://mlsanity.com/paper/2111.06464/) (2021)
- [Emergence of hierarchical reference systems in multi-agent communication](https://mlsanity.com/paper/2203.13176/) (2022)
- [Signal Shaping for Semantic Communication Systems with A Few Message Candidates](https://mlsanity.com/paper/2202.02072/) (2022)
- [Look-Ahead Task Offloading for Multi-User Mobile Augmented Reality in Edge-Cloud Computing](https://mlsanity.com/paper/2305.19558/) (2023)

## Abstract (arXiv)

Mobile augmented reality (MAR) is widely acknowledged as one of the ubiquitous interfaces to the digital twin and Metaverse, demanding unparalleled levels of latency, computational power, and energy efficiency. The existing solutions for realizing MAR combine multiple technologies like edge, cloud computing, and fifth-generation (5G) networks. However, the inherent communication latency of visual data imposes apparent limitations on the quality of experience (QoE). To address the challenge, we propose an emergent semantic communication framework to learn the communication protocols in MAR. Specifically, we train two agents through a modified Lewis signaling game to emerge a discrete communication protocol spontaneously. Based on this protocol, two agents can communicate about the abstract idea of visual data through messages with extremely small data sizes in a noisy channel, which leads to message errors. To better simulate real-world scenarios, we incorporate channel uncertainty into our training process. Experiments have shown that the proposed scheme has better generalization on unseen objects than traditional object recognition used in MAR and can effectively enhance communication efficiency through the utilization of small-size messages.