mlsanityData to 7 Oct 2026

Epistemic Neural Networks

arXiv 2107.08924 ↗First posted 19 Jul 2021cs.LG cs.AI stat.MLcited by 120

Not on the map. Why?

Abstract (arXiv)

Intelligence relies on an agent's knowledge of what it does not know. This capability can be assessed based on the quality of joint predictions of labels across multiple inputs. In principle, ensemble-based approaches produce effective joint predictions, but the computational costs of training large ensembles can become prohibitive. We introduce the epinet: an architecture that can supplement any conventional neural network, including large pretrained models, and can be trained with modest incremental computation to estimate uncertainty. With an epinet, conventional neural networks outperform very large ensembles, consisting of hundreds or more particles, with orders of magnitude less computation. The epinet does not fit the traditional framework of Bayesian neural networks. To accommodate development of approaches beyond BNNs, such as the epinet, we introduce the epistemic neural network (ENN) as an interface for models that produce joint predictions.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Builds on 48 of its 96 references that are in the map. First posted on arXiv 19 Jul 2021.

Builds on

48 of 96 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  12. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
36 more
  1. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  2. 2018Model Cards for Model Reportingcited by 1,068
  3. 2017Deep Bayesian Active Learning with Image Datacited by 812
  4. 2017Deep Neural Networks as Gaussian Processescited by 617
  5. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  6. 2015Weight Uncertainty in Neural Networkscited by 554
  7. 2016Deep Exploration via Bootstrapped DQNcited by 518
  8. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  9. 2017A Tutorial on Thompson Samplingcited by 420
  10. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  11. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  12. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  13. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  14. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  15. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 197
  16. 2020Training independent subnetworks for robust predictioncited by 177
  17. 2020Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Countscited by 163
  18. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  19. 2017Deep Exploration via Randomized Value Functionscited by 155
  20. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  21. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  22. 2019Behaviour Suite for Reinforcement Learningcited by 129
  23. 2019Functional Variational Bayesian Neural Networkscited by 127
  24. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  25. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  26. 2021On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertaintycited by 87
  27. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  28. 2020The Case for Bayesian Deep Learningcited by 68
  29. 2017Ensemble Samplingcited by 62
  30. 2021Reinforcement Learning, Bit by Bitcited by 59
  31. 2015Bootstrapped Thompson Sampling and Deep Explorationcited by 40
  32. 2020Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Modelscited by 39
  33. 2020Hypermodels for Explorationcited by 37
  34. 2021Correlated Input-Dependent Label Noise in Large-Scale Image Classificationcited by 37
  35. 2020Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?cited by 21
  36. 2017Variational Gaussian Dropout is not Bayesiancited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 19; 2023: 22; 2024: 36; 2025: 15; 2026: 23.

120 papers.

Sort:

By area:

  1. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  2. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  3. 2024To Believe or Not to Believe Your LLMcited by 120
  4. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  5. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  6. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  7. 2023Selectively Answering Ambiguous Questionscited by 63
  8. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  9. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  10. 2024Distinguishing the Knowable from the Unknowable with Language Modelscited by 45
  11. 2024Efficient Exploration for LLMscited by 44
  12. 2024Rethinking Aleatoric and Epistemic Uncertaintycited by 40
38 more
  1. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  2. 2024On the Challenges and Opportunities in Generative AIcited by 31
  3. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  4. 2022Regret Bounds for Information-Directed Reinforcement Learningcited by 24
  5. 2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
  6. 2021The Neural Testbed: Evaluating Joint Predictionscited by 22
  7. 2022Fine-Tuning Language Models via Epistemic Neural Networkscited by 22
  8. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  9. 2022Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrappingcited by 19
  10. 2023Random-Set Neural Networks (RS-NN)cited by 19
  11. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  12. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  13. 2024Experts Don't Cheat: Learning What You Don't Know By Predicting Pairscited by 18
  14. 2023Generalized Bayesian Inference for Scientific Simulators via Amortized Cost Estimationcited by 17
  15. 2023Fairness through Aleatoric Uncertaintycited by 16
  16. 2021Deep Exploration for Recommendation Systemscited by 14
  17. 2023Leveraging Demonstrations to Improve Online Learning: Quality Matterscited by 13
  18. 2024Scalable Bayesian Learning with posteriorscited by 13
  19. 2023Black-Box Batch Active Learning for Regressioncited by 12
  20. 2023Scalable Neural Contextual Bandit for Recommender Systemscited by 12
  21. 2023Massively Scaling Heteroscedastic Classifierscited by 11
  22. 2024The Role of Predictive Uncertainty and Diversity in Embodied AI and Robot Learningcited by 11
  23. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  24. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 11
  25. 2022Evaluating High-Order Predictive Distributions in Deep Learningcited by 10
  26. 2022Reinforcement Learning in Non-Markovian Environmentscited by 10
  27. 2024Composite Bayesian Optimization In Function Spaces Using NEON -- Neural Epistemic Operator Networkscited by 10
  28. 2024On the Importance of Uncertainty in Decision-Making with Large Language Modelscited by 9
  29. 2024A Structured Review of Literature on Uncertainty in Machine Learning & Deep Learningcited by 9
  30. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Modelscited by 8
  31. 2023Reducing LLM Hallucinations using Epistemic Neural Networkscited by 8
  32. 2024Provable Uncertainty Decomposition via Higher-Order Calibrationcited by 8
  33. 2025Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'cited by 8
  34. 2022Variational Neural Networkscited by 7
  35. 2022Uncertainty in Contrastive Learning: On the Predictability of Downstream Performancecited by 7
  36. 2023Probabilistic Circuits That Know What They Don't Knowcited by 7
  37. 2023Neural Exploitation and Exploration of Contextual Banditscited by 7
  38. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7

50 most cited of 120 All 120 (JSON)