mlsanityData to 7 Oct 2026

On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference

arXiv 2602.19964 ↗First posted 23 Feb 2026cs.LG cs.AI math.PRcited by 0

Not on the map. Why?

Abstract (arXiv)

Uncertainty quantification is central to safe and efficient deployments of deep learning models, yet many computationally practical methods lack lacking rigorous theoretical motivation. Random network distillation (RND) is a lightweight technique that measures novelty via prediction errors against a fixed random target. While empirically effective, it has remained unclear what uncertainties RND measures and how its estimates relate to other approaches, e.g. Bayesian inference or deep ensembles. This paper establishes these missing theoretical connections by analyzing RND within the neural tangent kernel framework in the limit of infinite network width. Our analysis reveals two central findings in this limit: (1) The uncertainty signal from RND -- its squared self-predictive error -- is equivalent to the predictive variance of a deep ensemble. (2) By constructing a specific RND target function, we show that the RND error distribution can be made to mirror the centered posterior predictive distribution of Bayesian inference with wide neural networks. Based on this equivalence, we moreover devise a posterior sampling algorithm that generates i.i.d. samples from an exact Bayesian posterior predictive distribution using this modified textitBayesian RND model. Collectively, our findings provide a unified theoretical perspective that places RND within the principled frameworks of deep ensembles and Bayesian inference, and offer new avenues for efficient yet theoretically grounded uncertainty quantification methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 57 references that are in the map. First posted on arXiv 23 Feb 2026.

Builds on

45 of 57 references.

References · 45

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016Variational Inference: A Review for Statisticianscited by 1,466
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2018Exploration by Random Network Distillationcited by 966
  7. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  8. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  11. 2017Deep Neural Networks as Gaussian Processescited by 617
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
33 more
  1. 2016Deep Exploration via Bootstrapped DQNcited by 518
  2. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  3. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  4. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  5. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  6. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  7. 2019Practical Deep Learning with Bayesian Principlescited by 158
  8. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  9. 2017Deep Exploration via Randomized Value Functionscited by 155
  10. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  11. 2020Bayesian Neural Networks: An Introduction and Surveycited by 130
  12. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  13. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
  14. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  15. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  16. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  17. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  18. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  19. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  20. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  21. 2023Anti-Exploration by Random Network Distillationcited by 49
  22. 2023Tractable Function-Space Variational Inference in Bayesian Neural Networkscited by 41
  23. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  24. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  25. 2021Exact Langevin Dynamics with Stochastic Gradientscited by 30
  26. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
  27. 2025Uncertainty Quantification with the Empirical Neural Tangent Kernelcited by 11
  28. 2023Diverse Projection Ensembles for Distributional Reinforcement Learningcited by 8
  29. 2022Disentangling the Predictive Variance of Deep Ensembles through the Neural Tangent Kernelcited by 4
  30. 2020On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architecturescited by 3
  31. 2024Epistemic Uncertainty and Observation Noise with the Neural Tangent Kernelcited by 3
  32. 2025Contextual Similarity Distillation: Ensemble Uncertainties with a Single Modelcited by 3
  33. 2025Universal Value-Function Uncertaintiescited by 1

Built on by

Not cited yet.