mlsanityData to 7 Oct 2026

Neural Discrete Representation Learning

arXiv 1711.00937 ↗First posted 2 Nov 2017cs.LGcited by 4,650

Not on the map. Why?

Cited by 799 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Learning useful representations without supervision remains a key challenge in machine learning. In this paper, we propose a simple yet powerful generative model that learns such discrete representations. Our model, the Vector Quantised-Variational AutoEncoder (VQ-VAE), differs from VAEs in two key ways: the encoder network outputs discrete, rather than continuous, codes; and the prior is learnt rather than static. In order to learn a discrete latent representation, we incorporate ideas from vector quantisation (VQ). Using the VQ method allows the model to circumvent issues of "posterior collapse" -- where the latents are ignored when they are paired with a powerful autoregressive decoder -- typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes, providing further evidence of the utility of the learnt representations.

Key facts. Cited by 4,650 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 248; 2022: 489; 2023: 728; 2024: 1044; 2025: 1239; 2026: 901.

4,650 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
38 more
  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  4. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  6. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  8. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  9. 2022Diffusion Models in Vision: A Surveycited by 1,008
  10. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  13. 2021Learning to Prompt for Continual Learningcited by 938
  14. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  15. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  16. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  17. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  18. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  19. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  20. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  21. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  22. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  23. 2024Genie: Generative Interactive Environmentscited by 706
  24. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  25. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  26. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  27. 2022High Fidelity Neural Audio Compressioncited by 673
  28. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  29. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  30. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  31. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  32. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  33. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  34. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  35. 2021Score-based Generative Modeling in Latent Spacecited by 634
  36. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  37. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  38. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611

50 most cited of 4,650 All 4,650 (JSON)