# Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning

arXiv: https://arxiv.org/abs/2505.07538 · HTML: https://mlsanity.com/paper/2505.07538/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 117 of its 130 references that are in the map.
- First posted on arXiv 12 May 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)

## Builds on (most cited first, 30 of 117)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)

## Built on by (most cited first, 30 of 33)

- [Show-o2: Improved Native Unified Multimodal Models](https://mlsanity.com/paper/2506.15564/) (2025)
- [NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale](https://mlsanity.com/paper/2508.10711/) (2025)
- [UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation](https://mlsanity.com/paper/2510.10575/) (2025)
- [IRBridge: Solving Image Restoration Bridge with Pre-trained Generative Diffusion Models](https://mlsanity.com/paper/2505.24406/) (2025)
- [WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editing](https://mlsanity.com/paper/2512.00387/) (2025)
- [Feedforward 3D Editing Learns from Semantic-Part Transformation](https://mlsanity.com/paper/2605.27351/) (2026)
- [Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learning](https://mlsanity.com/paper/2506.01480/) (2025)
- [WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction](https://mlsanity.com/paper/2508.05599/) (2025)
- [Reinforcement Learning for Large Model: A Survey](https://mlsanity.com/paper/2508.08189/) (2025)
- [Real-Time Motion-Controllable Autoregressive Video Diffusion](https://mlsanity.com/paper/2510.08131/) (2025)
- [CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenization](https://mlsanity.com/paper/2603.06449/) (2026)
- [VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation](https://mlsanity.com/paper/2608.18607/) (2026)
- [Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation](https://mlsanity.com/paper/2605.25195/) (2026)
- [Target-aware Image Editing via Cycle-consistent Constraints](https://mlsanity.com/paper/2510.20212/) (2025)
- [Plan-X: Instruct Video Generation via Semantic Planning](https://mlsanity.com/paper/2511.17986/) (2025)
- [WorldEdit: Towards Open-World Image Editing with a Knowledge-Informed Benchmark](https://mlsanity.com/paper/2602.07095/) (2026)
- [What Happens Next? Next Scene Prediction with a Unified Video Model](https://mlsanity.com/paper/2512.13015/) (2025)
- [Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation](https://mlsanity.com/paper/2603.16373/) (2026)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removal](https://mlsanity.com/paper/2508.07878/) (2025)
- [Expanding the Action Space of LLMs to Reason Beyond Language](https://mlsanity.com/paper/2510.07581/) (2025)
- [(1D) Ordered Tokens Enable Efficient Test-Time Search](https://mlsanity.com/paper/2604.15453/) (2026)
- [AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation](https://mlsanity.com/paper/2606.30811/) (2026)
- [UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2506.20214/) (2025)
- [On the Adversarial Robustness of Discrete Image Tokenizers](https://mlsanity.com/paper/2602.18252/) (2026)
- [Communication-Inspired Tokenization for Structured Image Representations](https://mlsanity.com/paper/2602.20731/) (2026)
- [Accelerating Diffusion Decoders via Multi-Scale Sampling and One-Step Distillation](https://mlsanity.com/paper/2603.19570/) (2026)
- [Autoregressive Visual Generation Needs a Prologue](https://mlsanity.com/paper/2605.06137/) (2026)
- [Learning Discrete Autoregressive Priors with Wasserstein Gradient Flow](https://mlsanity.com/paper/2605.06148/) (2026)
- [Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation](https://mlsanity.com/paper/2605.06207/) (2026)

## Abstract (arXiv)

We completely discard the conventional spatial prior in image representation and introduce a novel discrete visual tokenizer: Self-consistency Tokenizer (Selftok). At its design core, we compose an autoregressive (AR) prior -- mirroring the causal structure of language -- into visual tokens by using the reverse diffusion process of image generation. The AR property makes Selftok fundamentally distinct from traditional spatial tokens in the following two key ways: - Selftok offers an elegant and minimalist approach to unify diffusion and AR for vision-language models (VLMs): By representing images with Selftok tokens, we can train a VLM using a purely discrete autoregressive architecture -- like that in LLMs -- without requiring additional modules or training objectives. - We theoretically show that the AR prior satisfies the Bellman equation, whereas the spatial prior does not. Therefore, Selftok supports reinforcement learning (RL) for visual generation with effectiveness comparable to that achieved in LLMs. Besides the AR property, Selftok is also a SoTA tokenizer that achieves a favorable trade-off between high-quality reconstruction and compression rate. We use Selftok to build a pure AR VLM for both visual comprehension and generation tasks. Impressively, without using any text-image training pairs, a simple policy gradient RL working in the visual tokens can significantly boost the visual generation benchmark, surpassing all the existing models by a large margin. Therefore, we believe that Selftok effectively addresses the long-standing challenge that visual tokens cannot support effective RL. When combined with the well-established strengths of RL in LLMs, this brings us one step closer to realizing a truly multimodal LLM. Project Page: https://selftok-team.github.io/report/.