mlsanityData to 14 Oct 2026

HART: Efficient Visual Generation with Hybrid Autoregressive Transformer

arXiv 2410.10812 ↗First posted 14 Oct 2024cs.CV cs.AI cs.LGcited by 132

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We introduce Hybrid Autoregressive Transformer (HART), an autoregressive (AR) visual generation model capable of directly generating 1024x1024 images, rivaling diffusion models in image generation quality. Existing AR models face limitations due to the poor image reconstruction quality of their discrete tokenizers and the prohibitive training costs associated with generating 1024px images. To address these challenges, we present the hybrid tokenizer, which decomposes the continuous latents from the autoencoder into two components: discrete tokens representing the big picture and continuous tokens representing the residual components that cannot be represented by the discrete tokens. The discrete component is modeled by a scalable-resolution discrete AR model, while the continuous component is learned with a lightweight residual diffusion module with only 37M parameters. Compared with the discrete-only VAR tokenizer, our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K, leading to a 31% generation FID improvement from 7.85 to 5.38. HART also outperforms state-of-the-art diffusion models in both FID and CLIP score, with 4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourced at https://github.com/mit-han-lab/hart.

Key facts. Cited by 132 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 66 of its 76 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

66 of 76 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis10 Oct
  2. 2024DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation10 Oct
  3. 2024CAR: Controllable Autoregressive Modeling for Visual Generation7 Oct
  4. 2024LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding4 Oct
  5. 2024ControlAR: Controllable Image Generation with Autoregressive Models3 Oct
  6. 2024Emu3: Next-Token Prediction is All You Need27 Sep
  7. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation6 Sep
  8. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models17 Jun
  9. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modeling14 Jun
  10. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction3 Apr
  11. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment8 Mar
  12. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation27 Feb
2 more
  1. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment17 Oct
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 52

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
38 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  6. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  7. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  8. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  9. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  10. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  11. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  12. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  13. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  14. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  15. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  16. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  17. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  18. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  19. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  20. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  21. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  22. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  23. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  24. 2022MAGVIT: Masked Generative Video Transformercited by 346
  25. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  26. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 329
  27. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  28. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  29. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  30. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  31. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  32. 2024PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Modelscited by 141
  33. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 94
  34. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  35. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  36. 2024VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modelingcited by 42
  37. 2024STAR: Scale-wise Text-conditioned AutoRegressive image generationcited by 28
  38. 2024DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latentscited by 20

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 73; 2026: 42.

132 papers.

Sort:

By area:

  1. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 304
  2. 2024Taming Rectified Flow for Inversion and Editingcited by 204
  3. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  4. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  5. 2024Randomized Autoregressive Visual Generationcited by 127
  6. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 96
  7. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 64
  8. 2025Fractal Generative Modelscited by 46
  9. 2024Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Headscited by 44
  10. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  11. 2024Autoregressive Models in Vision: A Surveycited by 39
  12. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
38 more
  1. 2025FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruningcited by 34
  2. 2024Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficientcited by 33
  3. 2024Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesiscited by 33
  4. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 33
  5. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  6. 2024Next Patch Prediction for Autoregressive Visual Generationcited by 26
  7. 2025LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMscited by 24
  8. 2025FlexVAR: Flexible Visual Autoregressive Modeling without Residual Predictioncited by 23
  9. 2025Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoningcited by 23
  10. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  11. 2025Training-Free Text-Guided Image Editing with Visual Autoregressive Modelcited by 21
  12. 2025VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learningcited by 21
  13. 2025HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generationcited by 20
  14. 2025Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Modelcited by 20
  15. 2024Continuous Speculative Decoding for Autoregressive Image Generationcited by 19
  16. 2025PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Modelscited by 18
  17. 2025SCALAR: Scale-wise Controllable Visual Autoregressive Learningcited by 18
  18. 2025Visual Generation Without Guidancecited by 17
  19. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  20. 2025Soft-Masked Diffusion Language Modelscited by 16
  21. 2024X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Modelscited by 15
  22. 2025Privacy Attacks on Image AutoRegressive Modelscited by 15
  23. 2025The Power of Context: How Multimodality Improves Image Super-Resolutioncited by 15
  24. 2025RestoreVAR: Visual Autoregressive Generation for All-in-One Image Restorationcited by 15
  25. 2025Frequency-Aware Autoregressive Modeling for Efficient High-Resolution Image Synthesiscited by 15
  26. 2024REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latentscited by 14
  27. 2025Effective and Efficient Masked Image Generation Modelscited by 13
  28. 2025DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizercited by 13
  29. 2025Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropycited by 12
  30. 2025Scale-wise Distillation of Diffusion Modelscited by 11
  31. 2025Elucidating the Design Space of Multimodal Protein Language Modelscited by 11
  32. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  33. 2024Text-to-Image Synthesis: A Decade Surveycited by 9
  34. 2025D-AR: Diffusion via Autoregressive Modelscited by 9
  35. 2025MVAR: Visual Autoregressive Modeling with Scale and Spatial Markovian Conditioningcited by 8
  36. 2025Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivotscited by 8
  37. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  38. 2024Efficient Generative Modeling with Residual Vector Quantization-Based Tokenscited by 7

50 most cited of 132 All 132 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Universal Approximation of Visual Autoregressive Transformers
  2. 2024E-CAR: Efficient Continuous Autoregressive Image Generation via Multistage Modeling
  3. 2025Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregression
  4. 2024Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis
  5. 2023StraIT: Non-autoregressive Generation with Stratified Image Transformer
  6. 2025ARSS: Taming Decoder-only Autoregressive Visual Generation for View Synthesis From Single View
  7. 2025Scale-Wise VAR is Secretly Discrete Diffusion
  8. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL