# Discrete and continuous representations and processing in deep learning: Looking forward

arXiv: https://arxiv.org/abs/2201.01233 · HTML: https://mlsanity.com/paper/2201.01233/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jan 2022.

## Built on by (most cited first, 12 of 12)

- [Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review](https://mlsanity.com/paper/2202.12205/) (2022)
- [Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models](https://mlsanity.com/paper/2303.07735/) (2023)
- [Hierarchical Residual Learning Based Vector Quantized Variational Autoencoder for Image Reconstruction and Generation](https://mlsanity.com/paper/2208.04554/) (2022)
- [Learning to Play Atari in a World of Tokens](https://mlsanity.com/paper/2406.01361/) (2024)
- [Additive Models Explained: A Computational Complexity Approach](https://mlsanity.com/paper/2510.21292/) (2025)
- [Unconditional Image-Text Pair Generation with Multimodal Cross Quantizer](https://mlsanity.com/paper/2204.07537/) (2022)
- [Towards Equipping Transformer with the Ability of Systematic Compositionality](https://mlsanity.com/paper/2312.07280/) (2023)
- [Do Transformers know symbolic rules, and would we know if they did?](https://mlsanity.com/paper/2203.00162/) (2022)
- [Unbiased Weight Maximization](https://mlsanity.com/paper/2307.13270/) (2023)
- [Computational Natural Philosophy: A Thread from Presocratics through Turing to ChatGPT](https://mlsanity.com/paper/2309.13094/) (2023)
- [Privacy Attack in Federated Learning is Not Easy: An Experimental Study](https://mlsanity.com/paper/2409.19301/) (2024)
- [Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGI](https://mlsanity.com/paper/2511.18517/) (2025)

## Abstract (arXiv)

Discrete and continuous representations of content (e.g., of language or images) have interesting properties to be explored for the understanding of or reasoning with this content by machines. This position paper puts forward our opinion on the role of discrete and continuous representations and their processing in the deep learning field. Current neural network models compute continuous-valued data. Information is compressed into dense, distributed embeddings. By stark contrast, humans use discrete symbols in their communication with language. Such symbols represent a compressed version of the world that derives its meaning from shared contextual information. Additionally, human reasoning involves symbol manipulation at a cognitive level, which facilitates abstract reasoning, the composition of knowledge and understanding, generalization and efficient learning. Motivated by these insights, in this paper we argue that combining discrete and continuous representations and their processing will be essential to build systems that exhibit a general form of intelligence. We suggest and discuss several avenues that could improve current neural networks with the inclusion of discrete elements to combine the advantages of both types of representations.