mlsanityData to 14 Oct 2026

Monitizer: Automating Design and Evaluation of Neural Network Monitors

arXiv 2405.10350 ↗First posted 16 May 2024cs.LG cs.AI cs.SEcited by 2

Not on the map. Why?

Abstract (arXiv)

The behavior of neural networks (NNs) on previously unseen types of data (out-of-distribution or OOD) is typically unpredictable. This can be dangerous if the network's output is used for decision-making in a safety-critical system. Hence, detecting that an input is OOD is crucial for the safe application of the NN. Verification approaches do not scale to practical NNs, making runtime monitoring more appealing for practical use. While various monitors have been suggested recently, their optimization for a given problem, as well as comparison with each other and reproduction of results, remain challenging. We present a tool for users and developers of NN monitors. It allows for (i) application of various types of monitors from the literature to a given input NN, (ii) optimization of the monitor's hyperparameters, and (iii) experimental evaluation and comparison to other approaches. Besides, it facilitates the development of new monitoring approaches. We demonstrate the tool's usability on several use cases of different types of users as well as on a case study comparing different approaches from recent literature.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

39 of 57 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  10. 2013Describing Textures in the Wildcited by 2,125
  11. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  12. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
27 more
  1. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  2. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 600
  3. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  4. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  5. 2018Deep Learning for Classical Japanese Literaturecited by 422
  6. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  7. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 405
  8. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 339
  9. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 268
  10. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  11. 2021A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detectioncited by 207
  12. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 179
  13. 2021DICE: Leveraging Sparsification for Out-of-Distribution Detectioncited by 166
  14. 2016Lost and Found: Detecting Small Road Hazards for Self-Driving Vehiclescited by 117
  15. 2022Training OOD Detectors in their Natural Habitatscited by 86
  16. 2021PRIMA: General and Precise Neural Network Certification via Scalable Convex Hull Approximationscited by 77
  17. 2023Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detectioncited by 57
  18. 2022The Third International Verification of Neural Networks Competition (VNN-COMP 2022): Summary and Resultscited by 39
  19. 2019Outside the Box: Abstraction-Based Monitoring of Neural Networkscited by 36
  20. 2021Neural Network Robustness as a Verification Property: A Principled Case Studycited by 35
  21. 2018Runtime Monitoring Neuron Activation Patternscited by 34
  22. 2021No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasetscited by 31
  23. 2022Out-Of-Distribution Detection Is Not All You Needcited by 23
  24. 2019Towards Structured Evaluation of Deep Neural Network Supervisorscited by 19
  25. 2022Detection of out-of-distribution samples using binary neuron activation patternscited by 15
  26. 2021Benchmarking Safety Monitors for Image Classifiers with Machine Learningcited by 14
  27. 2022Runtime Monitoring for Out-of-Distribution Detection in Object Detection Neural Networkscited by 7

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Gaussian-Based and Outside-the-Box Runtime Monitoring Join Forcescited by 2
  2. 2025Benchmarking Vision Foundation Models for Input Monitoring in Autonomous Drivingcited by 2