# Disentanglement and Compositionality of Letter Identity and Letter Position in Variational Auto-Encoder Vision Models

arXiv: https://arxiv.org/abs/2412.10446 · HTML: https://mlsanity.com/paper/2412.10446/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 64 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 28 of 28)

- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Neural Networks and the Chomsky Hierarchy](https://mlsanity.com/paper/2207.02098/) (2022)
- [Why do deep convolutional networks generalize so poorly to small image transformations?](https://mlsanity.com/paper/1805.12177/) (2018)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://mlsanity.com/paper/2003.12294/) (2020)
- [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://mlsanity.com/paper/2108.09661/) (2021)
- [SVTR: Scene Text Recognition with a Single Visual Model](https://mlsanity.com/paper/2205.00159/) (2022)
- [RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition](https://mlsanity.com/paper/2007.07542/) (2020)
- [TextScanner: Reading Characters in Order for Robust Scene Text Recognition](https://mlsanity.com/paper/1912.12422/) (2019)
- [What If We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labels](https://mlsanity.com/paper/2103.04400/) (2021)
- [Visual Representation Learning Does Not Generalize Strongly Within the Same Domain](https://mlsanity.com/paper/2107.08221/) (2021)
- [Linguistic generalization and compositionality in modern artificial neural networks](https://mlsanity.com/paper/1904.00157/) (2019)
- [Contextual Position Encoding: Learning to Count What's Important](https://mlsanity.com/paper/2405.18719/) (2024)
- [MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining](https://mlsanity.com/paper/2206.00311/) (2022)
- [Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons](https://mlsanity.com/paper/2006.14304/) (2020)
- [Disentanglement with Biological Constraints: A Theory of Functional Cell Types](https://mlsanity.com/paper/2210.01768/) (2022)
- [Primitive Representation Learning for Scene Text Recognition](https://mlsanity.com/paper/2105.04286/) (2021)
- [Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognition](https://mlsanity.com/paper/2107.12090/) (2021)
- [Exploiting the Potential of Standard Convolutional Autoencoders for Image Restoration by Evolutionary Search](https://mlsanity.com/paper/1803.00370/) (2018)
- [Addressing the Topological Defects of Disentanglement via Distributed Operators](https://mlsanity.com/paper/2102.05623/) (2021)
- [Towards the Unseen: Iterative Text Recognition by Distilling from Errors](https://mlsanity.com/paper/2107.12081/) (2021)
- [AutoSTR: Efficient Backbone Search for Scene Text Recognition](https://mlsanity.com/paper/2003.06567/) (2020)
- [Representation and Correlation Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://mlsanity.com/paper/2106.06960/) (2021)

## Abstract (arXiv)

Human readers can accurately count how many letters are in a word (e.g., 7 in ``buffalo''), remove a letter from a given position (e.g., ``bufflo'') or add a new one. The human brain of readers must have therefore learned to disentangle information related to the position of a letter and its identity. Such disentanglement is necessary for the compositional, unbounded, ability of humans to create and parse new strings, with any combination of letters appearing in any positions. Do modern deep neural models also possess this crucial compositional ability? Here, we tested whether neural models that achieve state-of-the-art on disentanglement of features in visual input can also disentangle letter position and letter identity when trained on images of written words. Specifically, we trained beta variational autoencoder (β-VAE) to reconstruct images of letter strings and evaluated their disentanglement performance using CompOrth - a new benchmark that we created for studying compositional learning and zero-shot generalization in visual models for orthography. The benchmark suggests a set of tests, of increasing complexity, to evaluate the degree of disentanglement between orthographic features of written words in deep neural models. Using CompOrth, we conducted a set of experiments to analyze the generalization ability of these models, in particular, to unseen word length and to unseen combinations of letter identities and letter positions. We found that while models effectively disentangle surface features, such as horizontal and vertical `retinal' locations of words within an image, they dramatically fail to disentangle letter position and letter identity and lack any notion of word length. Together, this study demonstrates the shortcomings of state-of-the-art β-VAE models compared to humans and proposes a new challenge and a corresponding benchmark to evaluate neural models.