# Disentangling What and Where for 3D Object-Centric Representations Through Active Inference

arXiv: https://arxiv.org/abs/2108.11762 · HTML: https://mlsanity.com/paper/2108.11762/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [Motivating the Rules of the Game for Adversarial Example Research](https://mlsanity.com/paper/1807.06732/) (2018)
- [Information-Directed Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.07544/) (2018)
- [Encoders and Ensembles for Task-Free Continual Learning](https://mlsanity.com/paper/2105.13327/) (2021)

## Built on by (most cited first, 3 of 3)

- [Designing Ecosystems of Intelligence from First Principles](https://mlsanity.com/paper/2212.01354/) (2022)
- [Interpretable part-whole hierarchies and conceptual-semantic relationships in neural networks](https://mlsanity.com/paper/2203.03282/) (2022)
- [Disentangling Shape and Pose for Object-Centric Deep Active Inference Models](https://mlsanity.com/paper/2209.09097/) (2022)

## Abstract (arXiv)

Although modern object detection and classification models achieve high accuracy, these are typically constrained in advance on a fixed train set and are therefore not flexible to deal with novel, unseen object categories. Moreover, these models most often operate on a single frame, which may yield incorrect classifications in case of ambiguous viewpoints. In this paper, we propose an active inference agent that actively gathers evidence for object classifications, and can learn novel object categories over time. Drawing inspiration from the human brain, we build object-centric generative models composed of two information streams, a what- and a where-stream. The what-stream predicts whether the observed object belongs to a specific category, while the where-stream is responsible for representing the object in its internal 3D reference frame. We show that our agent (i) is able to learn representations for many object categories in an unsupervised way, (ii) achieves state-of-the-art classification accuracies, actively resolving ambiguity when required and (iii) identifies novel object categories. Furthermore, we validate our system in an end-to-end fashion where the agent is able to search for an object at a given pose from a pixel-based rendering. We believe that this is a first step towards building modular, intelligent systems that can be used for a wide range of tasks involving three dimensional objects.