mlsanityData to 7 Oct 2026

Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shift

arXiv 2409.03543 ↗First posted 5 Sep 2024cs.CV cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Natural distribution shift causes a deterioration in the perception performance of convolutional neural networks (CNNs). This comprehensive analysis for real-world traffic data addresses: 1) investigating the effect of natural distribution shift and weather augmentations on both detection quality and confidence estimation, 2) evaluating model performance for both classification and object localization, and 3) benchmarking two common uncertainty quantification methods - Ensembles and different variants of Monte-Carlo (MC) Dropout - under natural and close-to-natural distribution shift. For this purpose, a novel dataset has been curated from publicly available autonomous driving datasets. The in-distribution (ID) data is based on cutouts of a single object, for which both class and bounding box annotations are available. The six distribution-shift datasets cover adverse weather scenarios, simulated rain and fog, corner cases, and out-of-distribution data. A granular analysis of CNNs under distribution shift allows to quantize the impact of different types of shifts on both, task performance and confidence estimation: ConvNeXt-Tiny is more robust than EfficientNet-B0; heavy rain degrades classification stronger than localization, contrary to heavy fog; integrating MC-Dropout into selected layers only has the potential to enhance task performance and confidence estimation, whereby the identification of these layers depends on the type of distribution shift and the considered task.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 80 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

50 of 80 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2017Mask R-CNNcited by 7,259
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
38 more
  1. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  2. 2019Invariant Risk Minimizationcited by 1,832
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  4. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  5. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  6. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  7. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  8. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  9. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  10. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  11. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  12. 2019On Evaluating Adversarial Robustnesscited by 442
  13. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  14. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  15. 2021Objects are Different: Flexible Monocular 3D Object Detectioncited by 184
  16. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  17. 2018Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flowcited by 135
  18. 2020A Review and Comparative Study on Probabilistic Object Detection in Autonomous Drivingcited by 115
  19. 2020Hyper-Parameter Optimization: A Review of Algorithms and Applicationscited by 114
  20. 2018An introduction to domain adaptation and transfer learningcited by 112
  21. 2017YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Videocited by 109
  22. 2019Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Drivingcited by 105
  23. 2022A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglementcited by 87
  24. 2020Canadian Adverse Driving Conditions Datasetcited by 86
  25. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 85
  26. 2021Active Learning for Deep Object Detection via Probabilistic Modelingcited by 79
  27. 2021Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantificationcited by 57
  28. 2021OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Imagescited by 46
  29. 2021Near-Optimal Linear Regression under Distribution Shiftcited by 40
  30. 2019Uncertainty Estimation in One-Stage Object Detectioncited by 36
  31. 2022A new similarity measure for covariate shift with applications to nonparametric regressioncited by 33
  32. 2022The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcited by 25
  33. 2021Covariate Shift in High-Dimensional Random Feature Regressioncited by 24
  34. 2022MUAD: Multiple Uncertainties for Autonomous Driving, a benchmark for multiple uncertainty types and taskscited by 24
  35. 2023How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?cited by 14
  36. 2023A Benchmark on Uncertainty Quantification for Deep Learning Prognosticscited by 9
  37. 2021Benchmarking Uncertainty Quantification on Biosignal Classification Tasks under Dataset Shiftcited by 6
  38. 2023Overcoming the Limitations of Localization Uncertainty: Efficient & Exact Non-Linear Post-Processing and Calibrationcited by 5

Built on by

Not cited yet.