# Beyond Token-Level Cross-Entropy: Fr'echet Distributional Post-Training for Autoregressive Image Generation

arXiv: https://arxiv.org/abs/2608.00562 · HTML: https://mlsanity.com/paper/2608.00562/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 56 references that are in the map.
- First posted on arXiv 1 Aug 2026.

## Builds on (most cited first, 30 of 40)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [An Image is Worth 32 Tokens for Reconstruction and Generation](https://mlsanity.com/paper/2406.07550/) (2024)
- [Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets](https://mlsanity.com/paper/1903.05662/) (2019)
- [Professor Forcing: A New Algorithm for Training Recurrent Networks](https://mlsanity.com/paper/1610.09038/) (2016)
- [Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens](https://mlsanity.com/paper/2410.13863/) (2024)
- [MaskBit: Embedding-free Image Generation via Bit Tokens](https://mlsanity.com/paper/2409.16211/) (2024)
- [RandAR: Decoder-only Autoregressive Visual Generation in Random Orders](https://mlsanity.com/paper/2412.01827/) (2024)
- [FlowAR: Scale-wise Autoregressive Image Generation Meets Flow Matching](https://mlsanity.com/paper/2412.15205/) (2024)
- [GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation](https://mlsanity.com/paper/2504.08736/) (2025)
- [One-step Latent-free Image Generation with Pixel Mean Flows](https://mlsanity.com/paper/2601.22158/) (2026)
- [Scalable Image Tokenization with Index Backpropagation Quantization](https://mlsanity.com/paper/2412.02692/) (2024)
- [Representation Fr'echet Loss for Visual Generation](https://mlsanity.com/paper/2604.28190/) (2026)
- [Next Patch Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2412.15321/) (2024)
- [Autoregressive Image Generation with Randomized Parallel Decoding](https://mlsanity.com/paper/2503.10568/) (2025)
- [Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model](https://mlsanity.com/paper/2506.05289/) (2025)
- [Autoregressive Image Generation with Masked Bit Modeling](https://mlsanity.com/paper/2602.09024/) (2026)

## Built on by (most cited first, 1 of 1)

- [Efficient Fr'echet Distributional Decoder Alignment for Latent Generative Models](https://mlsanity.com/paper/2610.17466/) (2026)

## Abstract (arXiv)

Autoregressive image generators are commonly pretrained with token-level cross-entropy under teacher forcing, yet evaluated by the distributional quality of decoded images. This creates an objective mismatch, because categorical errors have unequal image-level consequences, and a context mismatch, because inference conditions on model-generated histories. We introduce FD-loss post-training, which adapts a pretrained discrete generator using representation-space Fr'echet distance as the sole objective. A dual-pass scheme first constructs detached rollout contexts through gradient-free generation under the model's native inference configuration, then performs differentiable replay with a probability-level straight-through estimator (STE) that preserves hard argmax decoding in the forward pass while propagating image-level gradients through temperature-scaled probabilities. Only the generator is updated, while the tokenizer and feature extractors remain frozen. Across eight completed configurations from four generator families on class-conditional ImageNet at 256×256, FD-loss post-training reduces FID and FDr6 by 41.4% and 52.0% on average. The strongest FID result improves from 2.42 to 1.43 without adding parameters or inference steps.