mlsanityData to 14 Oct 2026

Categorical Reparameterization with Gumbel-Softmax

arXiv 1611.01144 ↗First posted 3 Nov 2016stat.ML cs.LGcited by 2,739

Not on the map. Why?

Cited by 97 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Categorical variables are a natural choice for representing discrete structure in the world. However, stochastic neural networks rarely use categorical latent variables due to the inability to backpropagate through samples. In this work, we present an efficient gradient estimator that replaces the non-differentiable sample from a categorical distribution with a differentiable sample from a novel Gumbel-Softmax distribution. This distribution has the essential property that it can be smoothly annealed into a categorical distribution. We show that our Gumbel-Softmax estimator outperforms state-of-the-art gradient estimators on structured output prediction and unsupervised generative modeling tasks with categorical latent variables, and enables large speedups on semi-supervised classification.

Key facts. Cited by 2,739 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 4; 2019: 2; 2020: 2; 2021: 498; 2022: 477; 2023: 527; 2024: 430; 2025: 392; 2026: 407.

2,739 papers.

Sort:

By area:

  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2022Token Merging: Your ViT But Fastercited by 1,021
  4. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  7. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  8. 2022High Fidelity Neural Audio Compressioncited by 687
  9. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  10. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  11. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  12. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
38 more
  1. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  2. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  3. 2021Dynamic Neural Networks: A Surveycited by 408
  4. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 359
  5. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  6. 2019Causal Confusion in Imitation Learningcited by 288
  7. 2021On Explainability of Graph Neural Networks via Subgraph Explorationscited by 281
  8. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  9. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 252
  10. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  11. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  12. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  13. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  14. 2021D'ya like DAGs? A Survey on Structure Learning and Causal Discoverycited by 211
  15. 2023NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classificationcited by 210
  16. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  17. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
  18. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  19. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  20. 2023RADAR: Robust AI-Text Detection via Adversarial Learningcited by 190
  21. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
  22. 2021GraphDF: A Discrete Flow Model for Molecular Graph Generationcited by 180
  23. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
  24. 2022Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communicationscited by 175
  25. 2021Walk in the Cloud: Learning Curves for Point Clouds Shape Analysiscited by 172
  26. 2021Discrete Graph Structure Learning for Forecasting Multiple Time Seriescited by 171
  27. 2022Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanismcited by 171
  28. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  29. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  30. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
  31. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 160
  32. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  33. 2021GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimatorcited by 148
  34. 2021Unsupervised Speech Recognitioncited by 148
  35. 2021Illiterate DALL-E Learns to Composecited by 147
  36. 2022Weakly supervised causal representation learningcited by 140
  37. 2022Causality Inspired Representation Learning for Domain Generalizationcited by 138
  38. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134

50 most cited of 2,739 All 2,739 (JSON)