# Concept-Centric Token Interpretation for Vector-Quantized Generative Models

arXiv: https://arxiv.org/abs/2506.00698 · HTML: https://mlsanity.com/paper/2506.00698/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 44 references that are in the map.
- First posted on arXiv 31 May 2025.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [MAGVIT: Masked Generative Video Transformer](https://mlsanity.com/paper/2212.05199/) (2022)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis](https://mlsanity.com/paper/2211.09117/) (2022)
- [Interpreting CLIP's Image Representation via Text-Based Decomposition](https://mlsanity.com/paper/2310.05916/) (2023)
- [Synthesizing the preferred inputs for neurons in neural networks via deep generator networks](https://mlsanity.com/paper/1605.09304/) (2016)
- [Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks](https://mlsanity.com/paper/1602.03616/) (2016)
- [Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization](https://mlsanity.com/paper/2402.03161/) (2024)
- [Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization](https://mlsanity.com/paper/2309.04669/) (2023)
- [Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era](https://mlsanity.com/paper/2403.08946/) (2024)
- [Explaining a black-box using Deep Variational Information Bottleneck Approach](https://mlsanity.com/paper/1902.06918/) (2019)
- [Towards Explanation of DNN-based Prediction with Guided Feature Inversion](https://mlsanity.com/paper/1804.00506/) (2018)
- [How Well do Feature Visualizations Support Causal Understanding of CNN Activations?](https://mlsanity.com/paper/2106.12447/) (2021)
- [Unlocking Feature Visualization for Deeper Networks with MAgnitude Constrained Optimization](https://mlsanity.com/paper/2306.06805/) (2023)
- [End-to-End Saliency Mapping via Probability Distribution Prediction](https://mlsanity.com/paper/1804.01793/) (2018)

## Built on by (most cited first, 4 of 4)

- [RAU: Reference-based Anatomical Understanding with Vision Language Models](https://mlsanity.com/paper/2509.22404/) (2025)
- [Online Skill Learning for Web Agents via State-Grounded Dynamic Retrieval](https://mlsanity.com/paper/2606.04391/) (2026)
- [RSTR: Reducing SpatioTemporal Redundancy in Diffusion Transformers](https://mlsanity.com/paper/2512.14096/) (2025)
- [Self-Improving Small Object Grounding in LVLMs](https://mlsanity.com/paper/2606.01612/) (2026)

## Abstract (arXiv)

Vector-Quantized Generative Models (VQGMs) have emerged as powerful tools for image generation. However, the key component of VQGMs -- the codebook of discrete tokens -- is still not well understood, e.g., which tokens are critical to generate an image of a certain concept? This paper introduces Concept-Oriented Token Explanation (CORTEX), a novel approach for interpreting VQGMs by identifying concept-specific token combinations. Our framework employs two methods: (1) a sample-level explanation method that analyzes token importance scores in individual images, and (2) a codebook-level explanation method that explores the entire codebook to find globally relevant tokens. Experimental results demonstrate CORTEX's efficacy in providing clear explanations of token usage in the generative process, outperforming baselines across multiple pretrained VQGMs. Besides enhancing VQGMs transparency, CORTEX is useful in applications such as targeted image editing and shortcut feature detection. Our code is available at https://github.com/YangTianze009/CORTEX.