# Towards Emergent Language Symbolic Semantic Segmentation and Model Interpretability

arXiv: https://arxiv.org/abs/2007.09448 · HTML: https://mlsanity.com/paper/2007.09448/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2020.

## Built on by (most cited first, 9 of 9)

- [Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods](https://mlsanity.com/paper/2111.02398/) (2021)
- [DISSECT: Disentangled Simultaneous Explanations via Concept Traversals](https://mlsanity.com/paper/2105.15164/) (2021)
- [ProtoSeg: Interpretable Semantic Segmentation with Prototypical Parts](https://mlsanity.com/paper/2301.12276/) (2023)
- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Survey](https://mlsanity.com/paper/2405.01636/) (2024)
- [Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation](https://mlsanity.com/paper/2409.09497/) (2024)
- [Probabilistic and Semantic Descriptions of Image Manifolds and Their Applications](https://mlsanity.com/paper/2307.02881/) (2023)
- [MAProtoNet: A Multi-scale Attentive Interpretable Prototypical Part Network for 3D Magnetic Resonance Imaging Brain Tumor Classification](https://mlsanity.com/paper/2404.08917/) (2024)
- [Grounded Language Acquisition From Object and Action Imagery](https://mlsanity.com/paper/2309.06335/) (2023)

## Abstract (arXiv)

Recent advances in methods focused on the grounding problem have resulted in techniques that can be used to construct a symbolic language associated with a specific domain. Inspired by how humans communicate complex ideas through language, we developed a generalized Symbolic Semantic (S2) framework for interpretable segmentation. Unlike adversarial models (e.g., GANs), we explicitly model cooperation between two agents, a Sender and a Receiver, that must cooperate to achieve a common goal. The Sender receives information from a high layer of a segmentation network and generates a symbolic sentence derived from a categorical distribution. The Receiver obtains the symbolic sentences and co-generates the segmentation mask. In order for the model to converge, the Sender and Receiver must learn to communicate using a private language. We apply our architecture to segment tumors in the TCGA dataset. A UNet-like architecture is used to generate input to the Sender network which produces a symbolic sentence, and a Receiver network co-generates the segmentation mask based on the sentence. Our Segmentation framework achieved similar or better performance compared with state-of-the-art segmentation methods. In addition, our results suggest direct interpretation of the symbolic sentences to discriminate between normal and tumor tissue, tumor morphology, and other image characteristics.