# HSA: Hierarchical Slot Attention for Multi-granularity Scene-Decomposition

arXiv: https://arxiv.org/abs/2607.08249 · HTML: https://mlsanity.com/paper/2607.08249/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 46 of its 54 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos](https://mlsanity.com/paper/2206.07764/) (2022)
- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [Unsupervised Discovery of Object Radiance Fields](https://mlsanity.com/paper/2107.07905/) (2021)
- [Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation](https://mlsanity.com/paper/1503.00848/) (2015)
- [Object Scene Representation Transformer](https://mlsanity.com/paper/2206.06922/) (2022)
- [SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models](https://mlsanity.com/paper/2305.11281/) (2023)
- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)
- [Stacked Capsule Autoencoders](https://mlsanity.com/paper/1906.06818/) (2019)
- [Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities](https://mlsanity.com/paper/2306.04829/) (2023)
- [Improving Object-centric Learning with Query Optimization](https://mlsanity.com/paper/2210.08990/) (2022)
- [Learning Physical Graph Representations from Visual Scenes](https://mlsanity.com/paper/2006.12373/) (2020)
- [SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers](https://mlsanity.com/paper/2312.00648/) (2023)
- [Adaptive Slot Attention: Object Discovery with Dynamic Slot Number](https://mlsanity.com/paper/2406.09196/) (2024)
- [Temporally Consistent Object-Centric Learning by Contrasting Slots](https://mlsanity.com/paper/2412.14295/) (2024)

## Abstract (arXiv)

Slot attention is a powerful framework for object-centric learning, decomposing visual scenes into latent slots through iterative competitive attention. However, existing methods share two critical limitations: they decompose scenes into a flat set of slots at a single granularity, and this decomposition is based on appearance rather than semantics. Yet humans understand scenes through semantic hierarchies: separating foreground from background, recognizing object categories, and identifying individual instances. Crucially, such semantic hierarchies cannot emerge without supervision, because category names are human constructs, not visual patterns. We propose Hierarchical Slot Attention (HSA), which learns multi-granularity semantic scene decomposition from a single model. HSA decomposes scenes at three levels: holistic (foreground/background), semantic (object categories), and panoptic (individual instances). Using only 10% labeled data, combined with hierarchical alignment loss, HSA learns all three levels jointly. We further introduce grouping purity and containment to measure whether the hierarchy is encoded in representation space, not just output masks. Experiments on COCO and PASCAL VOC demonstrate that HSA outperforms the strongest flat baseline by up to +41.5 ARI at holistic, +14.6 at semantic, and +10.4 at panoptic level on COCO, with even larger gains on Pascal VOC, while requiring a single model instead of three. Code will be made available upon acceptance.