mlsanityData to 14 Oct 2026

The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables

arXiv 1611.00712 ↗First posted 2 Nov 2016cs.LG stat.MLcited by 1,148

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The reparameterization trick enables optimizing large scale stochastic computation graphs via gradient descent. The essence of the trick is to refactor each stochastic node into a differentiable function of its parameters and a random variable with fixed distribution. After refactoring, the gradients of the loss propagated by the chain rule through the graph are low variance unbiased estimators of the gradients of the expected loss. While many continuous random variables have such reparameterizations, discrete random variables lack useful reparameterizations due to the discontinuous nature of discrete states. In this work we introduce Concrete random variables---continuous relaxations of discrete random variables. The Concrete distribution is a new family of distributions with closed form densities and a simple reparameterization. Whenever a discrete stochastic node of a computation graph can be refactored into a one-hot bit representation that is treated continuously, Concrete stochastic nodes can be used with automatic differentiation to produce low-variance biased gradients of objectives (including objectives that depend on the log-probability of latent stochastic nodes) on the corresponding discrete graph. We demonstrate the effectiveness of Concrete relaxations on density estimation and structured prediction tasks using neural networks.

Key facts. Cited by 1,148 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 3; 2019: 2; 2020: 1; 2021: 233; 2022: 192; 2023: 209; 2024: 165; 2025: 151; 2026: 192.

1,148 papers.

Sort:

By area:

  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  4. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  5. 2019Causal Confusion in Imitation Learningcited by 288
  6. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 252
  7. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  8. 2021D'ya like DAGs? A Survey on Structure Learning and Causal Discoverycited by 211
  9. 2023NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classificationcited by 210
  10. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  11. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
  12. 2023RADAR: Robust AI-Text Detection via Adversarial Learningcited by 190
38 more
  1. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
  2. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
  3. 2022Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communicationscited by 175
  4. 2021Walk in the Cloud: Learning Curves for Point Clouds Shape Analysiscited by 172
  5. 2021Discrete Graph Structure Learning for Forecasting Multiple Time Seriescited by 171
  6. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 160
  7. 2025The Diffusion Dualitycited by 135
  8. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
  9. 2022GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoningcited by 128
  10. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  11. 2022Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecastingcited by 117
  12. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
  13. 2021DiBS: Differentiable Bayesian Structure Learningcited by 106
  14. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 106
  15. 2018Multi-Agent Generative Adversarial Imitation Learningcited by 100
  16. 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 100
  17. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  18. 2021The Bayesian Learning Rulecited by 99
  19. 2022SPECTRE: Spectral Conditioning Helps to Overcome the Expressivity Limits of One-shot Graph Generatorscited by 99
  20. 2021Implicit MLE: Backpropagating Through Discrete Exponential Family Distributionscited by 98
  21. 2022Deep End-to-end Causal Inferencecited by 95
  22. 2021Oops I Took A Gradient: Scalable Sampling for Discrete Distributionscited by 92
  23. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  24. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 87
  25. 2023Data Distillation: A Surveycited by 87
  26. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  27. 2023Modular Deep Learningcited by 83
  28. 2021A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learningcited by 82
  29. 2022An Introduction to Neural Data Compressioncited by 79
  30. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 79
  31. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 78
  32. 2022Deep Clustering: A Comprehensive Surveycited by 78
  33. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  34. 2022Controllable Natural Language Generation with Contrastive Prefixescited by 75
  35. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  36. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  37. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  38. 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71

50 most cited of 1,148 All 1,148 (JSON)