# GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation

arXiv: https://arxiv.org/abs/2504.08736 · HTML: https://mlsanity.com/paper/2504.08736/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 79 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)

## Built on by (most cited first, 30 of 57)

- [AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models](https://mlsanity.com/paper/2509.25162/) (2025)
- [AToken: A Unified Tokenizer for Vision](https://mlsanity.com/paper/2509.14476/) (2025)
- [Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer](https://mlsanity.com/paper/2510.06590/) (2025)
- [Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model](https://mlsanity.com/paper/2506.05289/) (2025)
- [Making Reconstruction FID Predictive of Diffusion Generation FID](https://mlsanity.com/paper/2603.05630/) (2026)
- [Image Tokenizer Needs Post-Training](https://mlsanity.com/paper/2509.12474/) (2025)
- [WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction](https://mlsanity.com/paper/2508.05599/) (2025)
- [MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation](https://mlsanity.com/paper/2511.18262/) (2025)
- [Improving Reconstruction of Representation Autoencoder](https://mlsanity.com/paper/2602.08620/) (2026)
- [HieraTok: Multi-Scale Visual Tokenizer Improves Image Reconstruction and Generation](https://mlsanity.com/paper/2509.23736/) (2025)
- [VA-π: Variational Policy Alignment for Pixel-Aware Autoregressive Generation](https://mlsanity.com/paper/2512.19680/) (2025)
- [EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation](https://mlsanity.com/paper/2603.12267/) (2026)
- [REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization](https://mlsanity.com/paper/2510.04450/) (2025)
- [DINO-Tok: Adapting DINO for Visual Tokenizers](https://mlsanity.com/paper/2511.20565/) (2025)
- [DeRA: Decoupled Representation Alignment for Video Tokenization](https://mlsanity.com/paper/2512.04483/) (2025)
- [Spherical Leech Quantization for Visual Tokenization and Generation](https://mlsanity.com/paper/2512.14697/) (2025)
- [Improving Flexible Image Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2601.01535/) (2026)
- [FACE: A Face-based Autoregressive Representation for High-Fidelity and Efficient Mesh Generation](https://mlsanity.com/paper/2603.01515/) (2026)
- [ResTok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2601.03955/) (2026)
- [End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer](https://mlsanity.com/paper/2605.00503/) (2026)
- [RecTok: Reconstruction Distillation along Rectified Flow](https://mlsanity.com/paper/2512.13421/) (2025)
- [Boosting Latent Diffusion Models via Disentangled Representation Alignment](https://mlsanity.com/paper/2601.05823/) (2026)
- [TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Models](https://mlsanity.com/paper/2601.08011/) (2026)
- [VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latents](https://mlsanity.com/paper/2602.04202/) (2026)
- [Adaptive 1D Video Diffusion Autoencoder](https://mlsanity.com/paper/2602.04220/) (2026)
- [Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation](https://mlsanity.com/paper/2604.02289/) (2026)
- [Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering](https://mlsanity.com/paper/2606.01911/) (2026)
- [Exploiting Discriminative Codebook Prior for Autoregressive Image Generation](https://mlsanity.com/paper/2508.10719/) (2025)
- [MacTok: Robust Continuous Tokenization for Image Generation](https://mlsanity.com/paper/2603.29634/) (2026)
- [Aligning Latent Geometry for Spherical Flow Matching in Image Generation](https://mlsanity.com/paper/2605.15193/) (2026)

## Abstract (arXiv)

In autoregressive (AR) image generation, visual tokenizers compress images into compact discrete latent tokens, enabling efficient training of downstream autoregressive models for visual generation via next-token prediction. While scaling visual tokenizers improves image reconstruction quality, it often degrades downstream generation quality -- a challenge not adequately addressed in existing literature. To address this, we introduce GigaTok, the first approach to simultaneously improve image reconstruction, generation, and representation learning when scaling visual tokenizers. We identify the growing complexity of latent space as the key factor behind the reconstruction vs. generation dilemma. To mitigate this, we propose semantic regularization, which aligns tokenizer features with semantically consistent features from a pre-trained visual encoder. This constraint prevents excessive latent space complexity during scaling, yielding consistent improvements in both reconstruction and downstream autoregressive generation. Building on semantic regularization, we explore three key practices for scaling tokenizers:(1) using 1D tokenizers for better scalability, (2) prioritizing decoder scaling when expanding both encoder and decoder, and (3) employing entropy loss to stabilize training for billion-scale tokenizers. By scaling to bf3 space billion parameters, GigaTok achieves state-of-the-art performance in reconstruction, downstream AR generation, and downstream AR representation quality.