mlsanityData to 7 Oct 2026

On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty

arXiv 2102.11409 ↗First posted 22 Feb 2021cs.LG stat.MLcited by 87

Not on the map. Why?

Abstract (arXiv)

Inducing point Gaussian process approximations are often considered a gold standard in uncertainty estimation since they retain many of the properties of the exact GP and scale to large datasets. A major drawback is that they have difficulty scaling to high dimensional inputs. Deep Kernel Learning (DKL) promises a solution: a deep feature extractor transforms the inputs over which an inducing point Gaussian process is defined. However, DKL has been shown to provide unreliable uncertainty estimates in practice. We study why, and show that with no constraints, the DKL objective pushes "far-away" data points to be mapped to the same features as those of training-set points. With this insight we propose to constrain DKL's feature extractor to approximately preserve distances through a bi-Lipschitz constraint, resulting in a feature space favorable to DKL. We obtain a model, DUE, which demonstrates uncertainty quality outperforming previous DKL and other single forward pass uncertainty methods, while maintaining the speed and accuracy of standard neural networks.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 27 of its 53 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

27 of 53 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2016Wide Residual Networkscited by 3,057
  7. 2017Improved Training of Wasserstein GANscited by 2,223
  8. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  10. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  11. 2016Deep Exploration via Bootstrapped DQNcited by 518
  12. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
15 more
  1. 2015Deep Kernel Learningcited by 440
  2. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  3. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 408
  4. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  5. 2018Invertible Residual Networkscited by 364
  6. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  7. 2014Scalable Variational Gaussian Process Classificationcited by 226
  8. 2018i-RevNet: Deep Invertible Networkscited by 137
  9. 2016Stochastic Variational Deep Kernel Learningcited by 92
  10. 2014Manifold Gaussian Processes for Regressioncited by 79
  11. 2019Rates of Convergence for Sparse Variational Gaussian Process Regressioncited by 62
  12. 2020Identifying Causal-Effect Inference Failure with Uncertainty-Aware Modelscited by 57
  13. 2017Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networkscited by 48
  14. 2020A case for new neural network smoothness constraintscited by 43
  15. 2015Latent Gaussian Processes for Distribution Estimation of Multivariate Categorical Datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 16; 2023: 18; 2024: 18; 2025: 17; 2026: 7.

87 papers.

Sort:

By area:

  1. 2021Epistemic Neural Networkscited by 120
  2. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  3. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  4. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  5. 2022DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognitioncited by 42
  6. 2021AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantificationcited by 31
  7. 2023Quantification of Uncertainty with Adversarial Modelscited by 28
  8. 2023Introducing an Improved Information-Theoretic Measure of Predictive Uncertaintycited by 28
  9. 2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
  10. 2022Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcitycited by 24
  11. 2022Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networkscited by 23
  12. 2021Reliable Graph Neural Networks for Drug Discovery Under Distributional Shiftcited by 22
38 more
  1. 2024A Geometric Explanation of the Likelihood OOD Detection Paradoxcited by 22
  2. 2024Uncertainty Estimation by Density Aware Evidential Deep Learningcited by 22
  3. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  4. 2022Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Predictioncited by 21
  5. 2024Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage?cited by 17
  6. 2023Mitigating Transformer Overconfidence via Lipschitz Regularizationcited by 15
  7. 2021Natural Posterior Network: Deep Bayesian Uncertainty for Exponential Family Distributionscited by 13
  8. 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learningcited by 12
  9. 2021The Peril of Popular Deep Learning Uncertainty Estimation Methodscited by 11
  10. 2023Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimationcited by 11
  11. 2023Uncertainty Estimation for Molecules: Desiderata and Methodscited by 11
  12. 2021Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Datacited by 10
  13. 2022Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picturecited by 10
  14. 2023Gaussian Latent Representations for Uncertainty Estimation using Mahalanobis Distance in Deep Classifierscited by 10
  15. 2025Uncertainty Estimation by Flexible Evidential Deep Learningcited by 10
  16. 2021Deep Classifiers with Label Noise Modeling and Distance Awarenesscited by 8
  17. 2022Generating and Detecting True Ambiguity: A Forgotten Danger in DNN Supervision Testingcited by 8
  18. 2024Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensemblescited by 8
  19. 2023Training, Architecture, and Prior for Deterministic Uncertainty Methodscited by 7
  20. 2024Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shiftscited by 7
  21. 2024Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemblecited by 7
  22. 2023Guided Deep Kernel Learningcited by 6
  23. 2025OCCUQ: Exploring Efficient Uncertainty Quantification for 3D Occupancy Predictioncited by 6
  24. 2024Geometric Transformation Uncertainty for Improving 3D Fetal Brain Pose Prediction from Freehand 2D Ultrasound Videoscited by 5
  25. 2022Batch-Ensemble Stochastic Neural Networks for Out-of-Distribution Detectioncited by 4
  26. 2022On Attacking Out-Domain Uncertainty Estimation in Deep Neural Networkscited by 4
  27. 2023TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimationcited by 4
  28. 2023Density Uncertainty Layers for Reliable Uncertainty Estimationcited by 4
  29. 2024Transitional Uncertainty with Layered Intermediate Predictionscited by 4
  30. 2024Uncertainty Estimation and Out-of-Distribution Detection for LiDAR Scene Semantic Segmentationcited by 4
  31. 2025Martingale Posterior Neural Networks for Fast Sequential Decision Makingcited by 4
  32. 2021Estimating Uncertainty For Vehicle Motion Prediction on Yandex Shifts Datasetcited by 3
  33. 2022Improving Generalization via Uncertainty Driven Perturbationscited by 3
  34. 2022Learning Representation for Bayesian Optimization with Collision-free Regularizationcited by 3
  35. 2022Uncertainty Calibration in Bayesian Neural Networks via Distance-Aware Priorscited by 3
  36. 2023Vecchia Gaussian Process Ensembles on Internal Representations of Deep Neural Networkscited by 3
  37. 2023To Predict or to Reject: Causal Effect Estimation with Uncertainty on Networked Datacited by 3
  38. 2024A Rate-Distortion View of Uncertainty Quantificationcited by 3

50 most cited of 87 All 87 (JSON)