# Recursive Neural Programs: Variational Learning of Image Grammars and Part-Whole Hierarchies

arXiv: https://arxiv.org/abs/2206.08462 · HTML: https://mlsanity.com/paper/2206.08462/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 57 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [Predictive Coding Approximates Backprop along Arbitrary Computation Graphs](https://mlsanity.com/paper/2006.04182/) (2020)
- [Deep active inference agents using Monte-Carlo methods](https://mlsanity.com/paper/2006.04176/) (2020)
- [Hypernetwork functional image representation](https://mlsanity.com/paper/1902.10404/) (2019)
- [Fashionable Modelling with Flux](https://mlsanity.com/paper/1811.01457/) (2018)
- [Don't Unroll Adjoint: Differentiating SSA-Form Programs](https://mlsanity.com/paper/1810.07951/) (2018)
- [Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems](https://mlsanity.com/paper/2205.01128/) (2022)
- [Disentangling images with Lie group transformations and sparse coding](https://mlsanity.com/paper/2012.12071/) (2020)
- [Active Predictive Coding Networks: A Neural Solution to the Problem of Learning Reference Frames and Part-Whole Hierarchies](https://mlsanity.com/paper/2201.08813/) (2022)

## Built on by (most cited first, 2 of 2)

- [Deep hybrid models: infer and plan in a dynamic world](https://mlsanity.com/paper/2402.10088/) (2024)
- [Dynamic planning in hierarchical active inference](https://mlsanity.com/paper/2402.11658/) (2024)

## Abstract (arXiv)

Human vision involves parsing and representing objects and scenes using structured representations based on part-whole hierarchies. Computer vision and machine learning researchers have recently sought to emulate this capability using capsule networks, reference frames and active predictive coding, but a generative model formulation has been lacking. We introduce Recursive Neural Programs (RNPs), which, to our knowledge, is the first neural generative model to address the part-whole hierarchy learning problem. RNPs model images as hierarchical trees of probabilistic sensory-motor programs that recursively reuse learned sensory-motor primitives to model an image within different reference frames, forming recursive image grammars. We express RNPs as structured variational autoencoders (sVAEs) for inference and sampling, and demonstrate parts-based parsing, sampling and one-shot transfer learning for MNIST, Omniglot and Fashion-MNIST datasets, demonstrating the model's expressive power. Our results show that RNPs provide an intuitive and explainable way of composing objects and scenes, allowing rich compositionality and intuitive interpretations of objects in terms of part-whole hierarchies.