mlsanityData to 7 Oct 2026

Runtime Monitoring Neuron Activation Patterns

arXiv 1809.06573 ↗First posted 18 Sep 2018cs.LG stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

For using neural networks in safety critical domains, it is important to know if a decision made by a neural network is supported by prior similarities in training. We propose runtime neuron activation pattern monitoring - after the standard training process, one creates a monitor by feeding the training data to the network again in order to store the neuron activation patterns in abstract form. In operation, a classification decision over an input is further supplemented by examining if a pattern similar (measured by Hamming distance) to the generated pattern is contained in the monitor. If the monitor does not contain any pattern similar to the generated pattern, it raises a warning that the decision is not based on the training data. Our experiments show that, by adjusting the similarity-threshold for activation patterns, the monitors can report a significant portion of misclassfications to be not supported by training with a small false-positive rate, when evaluated on a test set.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 6; 2023: 9; 2024: 6; 2025: 6; 2026: 2.

34 papers.

Sort:

  1. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  2. 2021How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Reviewcited by 36
  3. 2022Out-Of-Distribution Detection Is Not All You Needcited by 23
  4. 2022Detection of out-of-distribution samples using binary neuron activation patternscited by 15
  5. 2021Benchmarking Safety Monitors for Image Classifiers with Machine Learningcited by 14
  6. 2023SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agentscited by 9
  7. 2022Unifying Evaluation of Machine Learning Safety Monitorscited by 8
  8. 2022Runtime Monitoring for Out-of-Distribution Detection in Object Detection Neural Networkscited by 7
  9. 2025The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilitiescited by 7
  10. 2021Customizable Reference Runtime Monitoring of Neural Networks using Resolution Boxescited by 6
  11. 2023What, Indeed, is an Achievable Provable Guarantee for Learning-Enabled Safety Critical Systemscited by 6
  12. 2023A Low-cost Strategic Monitoring Approach for Scalable and Interpretable Error Detection in Deep Neural Networkscited by 6
22 more
  1. 2024Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forwardcited by 5
  2. 2021Monitoring Object Detection Abnormalities via Data-Label and Post-Algorithm Abstractionscited by 4
  3. 2022Evaluation of Runtime Monitoring for UAV Emergency Landingcited by 4
  4. 2023An investigation of challenges encountered when specifying training data and runtime monitors for safety critical ML applicationscited by 4
  5. 2023LUNA: A Model-Based Universal Analysis Framework for Large Language Modelscited by 4
  6. 2024Unveiling AI's Blind Spots: An Oracle for In-Domain, Out-of-Domain, and Adversarial Errorscited by 3
  7. 2024Monitizer: Automating Design and Evaluation of Neural Network Monitorscited by 2
  8. 2024Gaussian-Based and Outside-the-Box Runtime Monitoring Join Forcescited by 2
  9. 2024Edge Case Detection in Automated Driving: Methods, Challenges, and Future Directionscited by 2
  10. 2025Revisiting Out-of-Distribution Detection in Real-time Object Detection: From Benchmark Pitfalls to a New Mitigation Paradigmcited by 2
  11. 2025Runtime Safety Monitoring of Deep Neural Networks for Perception: A Surveycited by 2
  12. 2021Abstraction and Symbolic Execution of Deep Neural Networks with Bayesian Approximation of Hidden Featurescited by 1
  13. 2023Runtime Monitoring DNN-Based Perceptioncited by 1
  14. 2025A New Perspective On AI Safety Through Control Theory Methodologiescited by 1
  15. 2022Safe AI -- How is this Possible?cited by 0
  16. 2023Preventing Errors in Person Detection: A Part-Based Self-Monitoring Frameworkcited by 0
  17. 2023Detecting Out-of-distribution Objects Using Neuron Activation Patternscited by 0
  18. 2024Landscape of AI safety concerns -- A methodology to support safety assurance for AI-based autonomous systemscited by 0
  19. 2025Explaining Unreliable Perception in Automated Driving: A Fuzzy-based Monitoring Approachcited by 0
  20. 2025DeepProv: Behavioral Characterization and Repair of Neural Networks via Inference Provenance Graph Analysiscited by 0
  21. 2026Unifying Runtime Monitoring Approaches for Safety-Critical Machine Learning: Application to Vision-Based Landingcited by 0
  22. 2026Operational Range Bounding in Spectroscopy: A Safety Cage Framework for Machine Learning Modelscited by 0