mlsanityData to 14 Oct 2026

Discrete-Valued Neural Communication

arXiv 2107.02367 ↗First posted 6 Jul 2021cs.LG cs.AIcited by 39

Not on the map. Why?

Abstract (arXiv)

Deep learning has advanced from fully connected architectures to structured models organized into components, e.g., the transformer composed of positional elements, modular architectures divided into slots, and graph neural nets made up of nodes. In structured models, an interesting question is how to conduct dynamic and possibly sparse communication among the separate components. Here, we explore the hypothesis that restricting the transmitted information among components to discrete representations is a beneficial bottleneck. The motivating intuition is human language in which communication occurs through discrete symbols. Even though individuals have different understandings of what a "cat" is based on their specific experiences, the shared discrete token makes it possible for communication among individuals to be unimpeded by individual differences in internal representation. To discretize the values of concepts dynamically communicated among specialist components, we extend the quantization mechanism from the Vector-Quantized Variational Autoencoder to multi-headed discretization with shared codebooks and use it for discrete-valued neural communication (DVNC). Our experiments show that DVNC substantially improves systematic generalization in a variety of architectures -- transformers, modular architectures, and graph neural networks. We also show that the DVNC is robust to the choice of hyperparameters, making the method very useful in practice. Moreover, we establish a theoretical justification of our discretization process, proving that it has the ability to increase noise robustness and reduce the underlying dimensionality of the model.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 31 of its 59 references that are in the map. First posted on arXiv 6 Jul 2021.

Builds on

31 of 59 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  6. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  7. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  8. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  9. 2015Neural Module Networkscited by 376
  10. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  11. 2017A simple neural network module for relational reasoningcited by 346
  12. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
19 more
  1. 2019Recurrent Independent Mechanismscited by 219
  2. 2019Contrastive Learning of Structured World Modelscited by 209
  3. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
  4. 2016Robust Large Margin Deep Neural Networkscited by 118
  5. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  6. 2015Neural Programmer-Interpreterscited by 101
  7. 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
  8. 2017Visual Interaction Networkscited by 95
  9. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 85
  10. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  11. 2016Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activationscited by 54
  12. 2019Routing Networks and the Challenges of Modular and Compositional Computationcited by 50
  13. 2020Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systemscited by 48
  14. 2016Generalization Error of Invariant Classifierscited by 42
  15. 2021Conditional Neural Relational Inference for Interacting Systemscited by 28
  16. 2018Relational Forward Models for Multi-Agent Learningcited by 21
  17. 2021Transformers with Competitive Ensembles of Independent Mechanismscited by 19
  18. 2019Improving Coordination in Small-Scale Multi-Agent Deep Reinforcement Learning through Memory-driven Communicationcited by 18
  19. 2017Discovering objects and their relations from entangled scene representationscited by 12

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 12; 2023: 14; 2024: 9; 2025: 1.

39 papers.

Sort:

By area:

  1. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 116
  2. 2023How Does Information Bottleneck Help Deep Learning?cited by 93
  3. 2022Trustworthy Graph Neural Networks: Aspects, Methods and Trendscited by 89
  4. 2023Disentanglement via Latent Quantizationcited by 40
  5. 2022Neural Systematic Bindercited by 37
  6. 2021From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligencecited by 35
  7. 2023The Relational Bottleneck as an Inductive Bias for Efficient Abstractioncited by 27
  8. 2022Simplicial Embeddings in Self-Supervised Learning and Downstream Classificationcited by 24
  9. 2022Robustness Implies Generalization via Data-Dependent Generalization Boundscited by 23
  10. 2023Learning Invariant Molecular Representation in Latent Discrete Spacecited by 19
  11. 2022Discrete Key-Value Bottleneckcited by 18
  12. 2023Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddingscited by 15
27 more
  1. 2022Representation Learning in Deep RL via Discrete Information Bottleneckcited by 13
  2. 2023Towards More Human-like AI Communication: A Review of Emergent Communication Researchcited by 13
  3. 2022Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantizationcited by 11
  4. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 11
  5. 2022Stateful active facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learningcited by 10
  6. 2023Unlearning via Sparse Representationscited by 10
  7. 2024Towards Principled Representation Learning from Videos for Reinforcement Learningcited by 10
  8. 2024Mitigating Adversarial Perturbations for Deep Reinforcement Learning via Vector Quantizationcited by 6
  9. 2024Improving Discrete Optimisation Via Decoupled Straight-Through Estimatorcited by 5
  10. 2021Neurosymbolic Systems of Perception & Cognition: The Role of Attentioncited by 4
  11. 2023Reusable Slotwise Mechanismscited by 4
  12. 2024Balance of Number of Embedding and their Dimensions in Vector Quantizationcited by 4
  13. 2023SoK: Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learningcited by 3
  14. 2023Towards Equipping Transformer with the Ability of Systematic Compositionalitycited by 2
  15. 2024Symbolic Autoencoding for Self-Supervised Sequence Learningcited by 2
  16. 2024VQSynery: Robust Drug Synergy Prediction With Vector Quantization Mechanismcited by 2
  17. 2024Gaussian Mixture Vector Quantization with Aggregated Categorical Posteriorcited by 2
  18. 2023Discrete Messages Improve Communication Efficiency among Isolated Intelligent Agentscited by 1
  19. 2024Efficient Continual Learning for Small Language Models with a Discrete Key-Value Bottleneckcited by 1
  20. 2021Properties of Minimizing Entropycited by 0
  21. 2022Coordinating Policies Among Multiple Agents via an Intelligent Communication Channelcited by 0
  22. 2022On a Built-in Conflict between Deep Learning and Systematic Generalizationcited by 0
  23. 2022A Short Survey of Systematic Generalizationcited by 0
  24. 2023Self-Organising Neural Discrete Representation Learning `a la Kohonencited by 0
  25. 2023Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputscited by 0
  26. 2024Discrete Dictionary-based Decomposition Layer for Structured Representation Learningcited by 0
  27. 2025A Theoretical Analysis of Provable Compositional Generalization in Neural Networks: A Necessary and Sufficient Conditioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Emergence of Machine Language: Towards Symbolic Intelligence with Neural Networks
  2. 2022Emergent Quantized Communication
  3. 2023Joint Coding-Modulation for Digital Semantic Communications via Variational Autoencoder
  4. 2026Neural Dynamics as the Composition of Quantized Units
  5. 2025Channel-Aware Vector Quantization for Robust Semantic Communication on Discrete Channels
  6. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networks
  7. 2022Graph-based Neural Modules to Inspect Attention-based Architectures: A Position Paper
  8. 2026The Emergent Symbolic Structure of Artificial Neural Networks