# Slot Attention with Re-Initialization and Self-Distillation

arXiv: https://arxiv.org/abs/2507.23755 · HTML: https://mlsanity.com/paper/2507.23755/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 19 of 19)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos](https://mlsanity.com/paper/2206.07764/) (2022)
- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding](https://mlsanity.com/paper/2406.19389/) (2024)
- [ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/2111.10265/) (2021)
- [SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models](https://mlsanity.com/paper/2305.11281/) (2023)
- [Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities](https://mlsanity.com/paper/2306.04829/) (2023)
- [Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames](https://mlsanity.com/paper/2302.04973/) (2023)
- [Self-supervised Object-Centric Learning for Videos](https://mlsanity.com/paper/2310.06907/) (2023)
- [SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers](https://mlsanity.com/paper/2312.00648/) (2023)
- [Vector-Quantized Vision Foundation Models for Object-Centric Learning](https://mlsanity.com/paper/2502.20263/) (2025)
- [Multi-Scale Fusion for Object Representation](https://mlsanity.com/paper/2410.01539/) (2024)
- [Grouped Discrete Representation for Object-Centric Learning](https://mlsanity.com/paper/2411.02299/) (2024)

## Built on by (most cited first, 14 of 14)

- [Predicting Video Slot Attention Queries from Random Slot-Feature Pairs](https://mlsanity.com/paper/2508.01345/) (2025)
- [Smoothing Slot Attention Iterations and Recurrences](https://mlsanity.com/paper/2508.05417/) (2025)
- [From Vicious to Virtuous Cycles: Synergistic Representation Learning for Unsupervised Video Object-Centric Learning](https://mlsanity.com/paper/2602.03390/) (2026)
- [Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning](https://mlsanity.com/paper/2603.22758/) (2026)
- [Dual-State Slot Attention: Decoupling Appearance and Identity for Video Object-Centric Learning](https://mlsanity.com/paper/2606.12601/) (2026)
- [Object-Centric Vision Token Pruning for Vision Language Models](https://mlsanity.com/paper/2511.20439/) (2025)
- [A Hyperbolic Perspective on Hierarchical Structure in Object-Centric Scene Representations](https://mlsanity.com/paper/2603.14022/) (2026)
- [Unsupervised Structural Scene Decomposition via Foreground-Aware Slot Attention with Pseudo-Mask Guidance](https://mlsanity.com/paper/2512.02685/) (2025)
- [When Slots Compete: Slot Merging in Object-Centric Learning](https://mlsanity.com/paper/2603.11246/) (2026)
- [InfoGeo: Information-Theoretic Object-Centric Learning for Cross-View Generalizable UAV Geo-Localization](https://mlsanity.com/paper/2605.07099/) (2026)
- [Cycle Consistency in Video Object-Centric Learning](https://mlsanity.com/paper/2605.30211/) (2026)
- [Internalizing Temporal Consistency in Video Object-Centric Learning without Explicit Regularization](https://mlsanity.com/paper/2605.31508/) (2026)
- [HSA: Hierarchical Slot Attention for Multi-granularity Scene-Decomposition](https://mlsanity.com/paper/2607.08249/) (2026)
- [Does Attention-Guided Masking Really Help Object Discovery in Object-Centric Learning?](https://mlsanity.com/paper/2609.15187/) (2026)

## Abstract (arXiv)

Unlike popular solutions based on dense feature maps, Object-Centric Learning (OCL) represents visual scenes as sub-symbolic object-level feature vectors, termed slots, which are highly versatile for tasks involving visual modalities. OCL typically aggregates object superpixels into slots by iteratively applying competitive cross attention, known as Slot Attention, with the slots as the query. However, once initialized, these slots are reused naively, causing redundant slots to compete with informative ones for representing objects. This often results in objects being erroneously segmented into parts. Additionally, mainstream methods derive supervision signals solely from decoding slots into the input's reconstruction, overlooking potential supervision based on internal information. To address these issues, we propose Slot Attention with re-Initialization and self-Distillation (DIAS): i) We reduce redundancy in the aggregated slots and re-initialize extra aggregation to update the remaining slots; ii) We drive the bad attention map at the first aggregation iteration to approximate the good at the last iteration to enable self-distillation. Experiments demonstrate that DIAS achieves state-of-the-art on OCL tasks like object discovery and recognition, while also improving advanced visual prediction and reasoning. Our source code and model checkpoints are available on https://github.com/Genera1Z/DIAS.