# Representational Homomorphism Predicts and Improves Compositional Generalization In Transformer Language Model

arXiv: https://arxiv.org/abs/2601.18858 · HTML: https://mlsanity.com/paper/2601.18858/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [A Benchmark for Systematic Generalization in Grounded Language Understanding](https://mlsanity.com/paper/2003.05161/) (2020)
- [A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers](https://mlsanity.com/paper/2503.03961/) (2025)
- [Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks](https://mlsanity.com/paper/2109.15256/) (2021)
- [A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practices](https://mlsanity.com/paper/2406.08787/) (2024)
- [Compositional Processing Emerges in Neural Networks Solving Math Problems](https://mlsanity.com/paper/2105.08961/) (2021)
- [Structural generalization in COGS: Supertagging is (almost) all you need](https://mlsanity.com/paper/2310.14124/) (2023)
- [Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers](https://mlsanity.com/paper/2310.12118/) (2023)
- [Systematic Generalization in Language Models Scales with Information Entropy](https://mlsanity.com/paper/2505.13089/) (2025)

## Built on by (most cited first, 2 of 2)

- [Adversarial Concept Search: Predicting Compositional Errors From Feature Geometry](https://mlsanity.com/paper/2606.13934/) (2026)
- [A Calibrated Test of Internal Action Maps: State Signals Without Global Affine Closure](https://mlsanity.com/paper/2608.13626/) (2026)

## Abstract (arXiv)

Compositional generalization-the ability to interpret novel combinations of familiar components-remains a persistent challenge for neural networks. Behavioral evaluations reveal when models fail but offer limited insight into why failures arise at the representational level. We introduce textitHomomorphism Error (HE), a structural metric that measures the inconsistency between a set of established rules for which words combine to form new meaning (linguistic syntax) and model's learned rules for which hidden states combine to form new states (semantic syntax). We formulate this inconsistency as deviations from approximate homomorphisms between the linguistic expression algebra and a model's hidden-state space. We designed experiments to test if i) HE predicts compositional generalization performance, and ii) will regularizing for low HE during training improve such performance. To avoid the effect of data spoilage, we train small decoder-only Transformers from scratch using an adapted version of established dataset, SCAN, for testing compositional generalization. Across controlled experiments, HE predicts out-of-distribution (OOD) compositional generalization under noise injection, achieving R2=0.73 correlation between HE and OOD accuracy. Ablations show that model depth has minimal effect on either HE or OOD accuracy, training data coverage exhibits threshold effects, and randomly inserted noise tokens increase HE. Intervention experiment shows that HE-regularized training significantly reduces HE (p=1.1×10-4) and yields a statistically significant improvement in OOD accuracy (p=0.023). Together, these results indicate the potential of HE to be both a diagnostic and an actionable training signal for improving compositional generalization.