mlsanityData to 7 Oct 2026

GIVT: Generative Infinite-Vocabulary Transformers

arXiv 2312.02116 ↗First posted 4 Dec 2023cs.CVcited by 80

Not on the map. Why?

Cited by 21 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Generative Infinite-Vocabulary Transformers (GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose two surprisingly simple modifications to decoder-only transformers: 1) at the input, we replace the finite-vocabulary lookup table with a linear projection of the input vectors; and 2) at the output, we replace the logits prediction (usually mapped to a categorical distribution) with the parameters of a multivariate Gaussian mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT, where transformers are used to model the discrete latent sequences of a VQ-VAE, we use GIVT to model the unquantized real-valued latent sequences of a β-VAE. In class-conditional image generation GIVT outperforms VQ-GAN (and improved variants thereof) as well as MaskGIT, and achieves performance competitive with recent latent diffusion models. Finally, we obtain strong results outside of image generation when applying GIVT to panoptic segmentation and depth estimation with a VAE variant of the UViM framework.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 68 of its 82 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

68 of 82 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
38 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  2. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  3. 2016Density estimation using Real NVPcited by 2,029
  4. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  5. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  9. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  10. 2021Scaling Vision Transformerscited by 993
  11. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  12. 2016Pixel Recurrent Neural Networkscited by 814
  13. 2018Panoptic Segmentationcited by 796
  14. 2018Image Transformercited by 770
  15. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 748
  16. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  17. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  18. 2023A decoder-only foundation model for time-series forecastingcited by 650
  19. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  20. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  21. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  22. 2020Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modulescited by 570
  23. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  24. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  25. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  26. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  27. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  28. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  29. 2018Assessing Generative Models via Precision and Recallcited by 401
  30. 2017VAE with a VampPriorcited by 247
  31. 2019Large Scale Adversarial Representation Learningcited by 238
  32. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  33. 2016Variational Lossy Autoencodercited by 175
  34. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  35. 2023TimeGPT-1cited by 149
  36. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  37. 2018i-RevNet: Deep Invertible Networkscited by 137
  38. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 40; 2026: 13.

80 papers.

Sort:

By area:

  1. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  2. 2024MaskBit: Embedding-free Image Generation via Bit Tokenscited by 96
  3. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 93
  4. 2024Normalizing Flows are Capable Generative Modelscited by 89
  5. 2025EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modelingcited by 85
  6. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  7. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  8. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 55
  9. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 55
  10. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 47
  11. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 44
  12. 2024DINO-Foresight: Looking into the Future with DINOcited by 44
38 more
  1. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 41
  2. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  3. 2025STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesiscited by 31
  4. 2025Bridging Continuous and Discrete Tokens for Autoregressive Visual Generationcited by 29
  5. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  6. 2024Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspectivecited by 22
  7. 2025Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminatorcited by 22
  8. 2024MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modelingcited by 19
  9. 2024Continuous Speculative Decoding for Autoregressive Image Generationcited by 18
  10. 2025Latent Denoising Makes Good Tokenizerscited by 18
  11. 2025FARMER: Flow AutoRegressive Transformer over Pixelscited by 17
  12. 2025Gaussian Mixture Flow Matching Modelscited by 16
  13. 2025Hyperspherical Latents Improve Continuous-Token Autoregressive Generationcited by 16
  14. 2025Watermarking Autoregressive Image Generationcited by 15
  15. 2024When Worse is Better: Navigating the compression-generation tradeoff in visual tokenizationcited by 14
  16. 2025Zero-Shot Styled Text Image Generation, but Make It Autoregressivecited by 13
  17. 2025Adapting Self-Supervised Representations as a Latent Space for Efficient Generationcited by 13
  18. 2024Denoising with a Joint-Embedding Predictive Architecturecited by 12
  19. 2024Continuous Autoregressive Models with Noise Augmentation Avoid Error Accumulationcited by 12
  20. 2024[MASK] is All You Needcited by 11
  21. 2025OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time Scalescited by 11
  22. 2025Image Tokenizer Needs Post-Trainingcited by 10
  23. 2025Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesiscited by 9
  24. 2025Continuous Visual Autoregressive Generation via Score Maximizationcited by 9
  25. 2025Continuous Autoregressive Language Modelscited by 9
  26. 2026Condition Errors Refinement in Autoregressive Image Generation with Diffusion Losscited by 9
  27. 2024Divot: Diffusion Powers Video Tokenizer for Comprehension and Generationcited by 8
  28. 2025Deeply Supervised Flow-Based Generative Modelscited by 8
  29. 2025Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivotscited by 8
  30. 2024Efficient Generative Modeling with Residual Vector Quantization-Based Tokenscited by 7
  31. 2025Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformerscited by 7
  32. 2024BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilitiescited by 5
  33. 2024Speech Synthesis From Continuous Features Using Per-Token Latent Diffusioncited by 5
  34. 2024DiCoDe: Diffusion-Compressed Deep Tokens for Autoregressive Video Generation with Language Modelscited by 5
  35. 2025Quantization-Free Autoregressive Action Transformercited by 5
  36. 2025Plug-and-Play Context Feature Reuse for Efficient Masked Generationcited by 5
  37. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Predictioncited by 5
  38. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 5

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LaVin-DiT: Large Vision Diffusion Transformer
  2. 2021VideoGPT: Video Generation using VQ-VAE and Transformers
  3. 2025Foundations and Models in Modern Computer Vision: Key Building Blocks in Landmark Architectures
  4. 2024Bag of Design Choices for Inference of High-Resolution Masked Generative Transformer
  5. 2024Comparative Analysis of Generative Models: Enhancing Image Synthesis with VAEs, GANs, and Stable Diffusion
  6. 2025Generative AI for Vision: A Comprehensive Study of Frameworks and Applications
  7. 2025GriDiT: Factorized Grid-Based Diffusion for Efficient Long Image Sequence Generation
  8. 2022Deep Generative Models on 3D Representations: A Survey