# Visual Concepts Tokenization

arXiv: https://arxiv.org/abs/2205.10093 · HTML: https://mlsanity.com/paper/2205.10093/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 20 May 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning](https://mlsanity.com/paper/2104.03135/) (2021)
- [Improving the Robustness of Deep Neural Networks via Stability Training](https://mlsanity.com/paper/1604.04326/) (2016)
- [On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset](https://mlsanity.com/paper/1906.03292/) (2019)
- [Unsupervised Learning of Compositional Energy Concepts](https://mlsanity.com/paper/2111.03042/) (2021)
- [Learning Disentangled Representation by Exploiting Pretrained Generative Models: A Contrastive Learning View](https://mlsanity.com/paper/2102.10543/) (2021)
- [Towards Building A Group-based Unsupervised Representation Disentanglement Framework](https://mlsanity.com/paper/2102.10303/) (2021)

## Built on by (most cited first, 14 of 14)

- [Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2311.13165/) (2023)
- [Towards Semantic Equivalence of Tokenization in Multimodal LLM](https://mlsanity.com/paper/2406.05127/) (2024)
- [Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement](https://mlsanity.com/paper/2402.09712/) (2024)
- [3D Point Cloud Pre-training with Knowledge Distillation from 2D Images](https://mlsanity.com/paper/2212.08974/) (2022)
- [Vector-based Representation is the Key: A Study on Disentanglement and Compositional Generalization](https://mlsanity.com/paper/2305.18063/) (2023)
- [Disentangling Disentangled Representations: Towards Improved Latent Units via Diffusion Models](https://mlsanity.com/paper/2410.23820/) (2024)
- [Concept-Centric Token Interpretation for Vector-Quantized Generative Models](https://mlsanity.com/paper/2506.00698/) (2025)
- [Learning Concept-Based Causal Transition and Symbolic Reasoning for Visual Planning](https://mlsanity.com/paper/2310.03325/) (2023)
- [Generating by Understanding: Neural Visual Generation with Logical Symbol Groundings](https://mlsanity.com/paper/2310.17451/) (2023)
- [Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding](https://mlsanity.com/paper/2403.18593/) (2024)
- [Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformers](https://mlsanity.com/paper/2411.15397/) (2024)
- [QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining](https://mlsanity.com/paper/2505.23004/) (2025)
- [AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens](https://mlsanity.com/paper/2606.07185/) (2026)
- [LinSlot: Exploiting Linear Representation hypothesis for unsupervised attribute discovery from slot based object representation](https://mlsanity.com/paper/2610.10722/) (2026)

## Abstract (arXiv)

Obtaining the human-like perception ability of abstracting visual concepts from concrete pixels has always been a fundamental and important target in machine learning research fields such as disentangled representation learning and scene decomposition. Towards this goal, we propose an unsupervised transformer-based Visual Concepts Tokenization framework, dubbed VCT, to perceive an image into a set of disentangled visual concept tokens, with each concept token responding to one type of independent visual concept. Particularly, to obtain these concept tokens, we only use cross-attention to extract visual information from the image tokens layer by layer without self-attention between concept tokens, preventing information leakage across concept tokens. We further propose a Concept Disentangling Loss to facilitate that different concept tokens represent independent visual concepts. The cross-attention and disentangling loss play the role of induction and mutual exclusion for the concept tokens, respectively. Extensive experiments on several popular datasets verify the effectiveness of VCT on the tasks of disentangled representation learning and scene decomposition. VCT achieves the state of the art results by a large margin.