mlsanityData to 14 Oct 2026

Heptapod: Language Modeling on Visual Signals

arXiv 2510.06673 ↗First posted 8 Oct 2025cs.CV cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We introduce Heptapod, an image autoregressive model that adheres to the foundational principles of language modeling. Heptapod employs causal attention, eliminates reliance on CFG, and eschews the trend of semantic tokenizers. Our key innovation is textitnext 2D distribution prediction: a causal Transformer with reconstruction-focused visual tokenizer, learns to predict the distribution over the entire 2D spatial grid of images at each timestep. This learning objective unifies the sequential modeling of autoregressive framework with the holistic self-supervised learning of masked autoencoding, enabling the model to capture comprehensive image semantics via generative training. On the ImageNet generation benchmark, Heptapod achieves an FID of 2.70, significantly outperforming previous causal autoregressive approaches. We hope our work inspires a principled rethinking of language modeling on visual signals and beyond.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 51 of its 60 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

51 of 60 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2022Scalable Diffusion Models with Transformerscited by 6,054
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2017Neural Discrete Representation Learningcited by 4,712
38 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2016Improved Techniques for Training GANscited by 2,970
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2021Variational Diffusion Modelscited by 1,153
  6. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  7. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  8. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  9. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
  10. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  11. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  12. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  13. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  14. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  15. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  16. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 393
  17. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  18. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  19. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 304
  20. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  21. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformerscited by 225
  22. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  23. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  24. 2024Randomized Autoregressive Visual Generationcited by 127
  25. 2022Generative Spoken Dialogue Language Modelingcited by 126
  26. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  27. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  28. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  29. 2023Textually Pretrained Speech Language Modelscited by 101
  30. 2024MaskBit: Embedding-free Image Generation via Bit Tokenscited by 98
  31. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 96
  32. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 94
  33. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 80
  34. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  35. 2025Fractal Generative Modelscited by 46
  36. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  37. 2024Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspectivecited by 22
  38. 2024When Worse is Better: Navigating the compression-generation tradeoff in visual tokenizationcited by 15

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codeccited by 2