# How to represent part-whole hierarchies in a neural network

arXiv: https://arxiv.org/abs/2102.12627 · HTML: https://mlsanity.com/paper/2102.12627/

## Key facts

- Cited by 123 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [Using Fast Weights to Attend to the Recent Past](https://mlsanity.com/paper/1610.06258/) (2016)
- [NASA: Neural Articulated Shape Approximation](https://mlsanity.com/paper/1912.03207/) (2019)
- [Stacked Capsule Autoencoders](https://mlsanity.com/paper/1906.06818/) (2019)
- [Grammar as a Foreign Language](https://mlsanity.com/paper/1412.7449/) (2014)
- [Learning Physical Graph Representations from Visual Scenes](https://mlsanity.com/paper/2006.12373/) (2020)
- [ISD: Self-Supervised Learning by Iterative Similarity Distillation](https://mlsanity.com/paper/2012.09259/) (2020)
- [Canonical Capsules: Self-Supervised Capsules in Canonical Pose](https://mlsanity.com/paper/2012.04718/) (2020)
- [Unsupervised part representation by Flow Capsules](https://mlsanity.com/paper/2011.13920/) (2020)
- [Geometric Capsule Autoencoders for 3D Point Clouds](https://mlsanity.com/paper/1912.03310/) (2019)

## Built on by (most cited first, 30 of 123)

- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Mobile-Former: Bridging MobileNet and Transformer](https://mlsanity.com/paper/2108.05895/) (2021)
- [Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs](https://mlsanity.com/paper/2203.06717/) (2022)
- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders](https://mlsanity.com/paper/2206.10207/) (2022)
- [Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors](https://mlsanity.com/paper/2103.15949/) (2021)
- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.12126/) (2022)
- [White-Box Transformers via Sparse Rate Reduction](https://mlsanity.com/paper/2306.01129/) (2023)
- [Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain](https://mlsanity.com/paper/2108.08487/) (2021)
- [TermiNeRF: Ray Termination Prediction for Efficient Neural Rendering](https://mlsanity.com/paper/2111.03643/) (2021)
- [A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges](https://mlsanity.com/paper/2112.15424/) (2021)
- [Hyperbolic Contrastive Learning for Visual Representations beyond Objects](https://mlsanity.com/paper/2212.00653/) (2022)
- [Concept Decomposition for Visual Exploration and Inspiration](https://mlsanity.com/paper/2305.18203/) (2023)
- [Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames](https://mlsanity.com/paper/2302.04973/) (2023)
- [Recovery of Continuous 3D Refractive Index Maps from Discrete Intensity-Only Measurements using Neural Fields](https://mlsanity.com/paper/2112.00002/) (2021)
- [Provably Learning Object-Centric Representations](https://mlsanity.com/paper/2305.14229/) (2023)
- [Neither hype nor gloom do DNNs justice](https://mlsanity.com/paper/2312.05355/) (2023)
- [Understanding Masked Autoencoders via Hierarchical Latent Variable Models](https://mlsanity.com/paper/2306.04898/) (2023)
- [Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding](https://mlsanity.com/paper/2208.00281/) (2022)
- [PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning](https://mlsanity.com/paper/2112.05136/) (2021)
- [Reversible Column Networks](https://mlsanity.com/paper/2212.11696/) (2022)
- [HGFormer: Hierarchical Grouping Transformer for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2305.13031/) (2023)
- [PartGlot: Learning Shape Part Segmentation from Language Reference Games](https://mlsanity.com/paper/2112.06390/) (2021)
- [Rotating Features for Object Discovery](https://mlsanity.com/paper/2306.00600/) (2023)
- [Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning](https://mlsanity.com/paper/2203.09064/) (2022)
- [GraphFPN: Graph Feature Pyramid Network for Object Detection](https://mlsanity.com/paper/2108.00580/) (2021)
- [Emergence of Segmentation with Minimalistic White-Box Transformers](https://mlsanity.com/paper/2308.16271/) (2023)

## Abstract (arXiv)

This paper does not describe a working system. Instead, it presents a single idea about representation which allows advances made by several different groups to be combined into an imaginary system called GLOM. The advances include transformers, neural fields, contrastive representation learning, distillation and capsules. GLOM answers the question: How can a neural network with a fixed architecture parse an image into a part-whole hierarchy which has a different structure for each image? The idea is simply to use islands of identical vectors to represent the nodes in the parse tree. If GLOM can be made to work, it should significantly improve the interpretability of the representations produced by transformer-like systems when applied to vision or language