# Robust Machine Learning Systems: Challenges, Current Trends, Perspectives, and the Road Ahead

arXiv: https://arxiv.org/abs/2101.02559 · HTML: https://mlsanity.com/paper/2101.02559/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 188 references that are in the map.
- First posted on arXiv 4 Jan 2021.

## Builds on (most cited first, 30 of 74)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://mlsanity.com/paper/1702.01135/) (2017)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [Certified Defenses for Data Poisoning Attacks](https://mlsanity.com/paper/1706.03691/) (2017)
- [Fooling automated surveillance cameras: adversarial patches to attack person detection](https://mlsanity.com/paper/1904.08653/) (2019)
- [Safety Verification of Deep Neural Networks](https://mlsanity.com/paper/1610.06940/) (2016)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Efficient Decision-based Black-box Adversarial Attacks on Face Recognition](https://mlsanity.com/paper/1904.04433/) (2019)
- [Efficient Formal Safety Analysis of Neural Networks](https://mlsanity.com/paper/1809.08098/) (2018)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [Query-Efficient Hard-label Black-box Attack:An Optimization-based Approach](https://mlsanity.com/paper/1807.04457/) (2018)

## Built on by (most cited first, 30 of 41)

- [I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences](https://mlsanity.com/paper/2206.08451/) (2022)
- [Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey](https://mlsanity.com/paper/2203.08737/) (2022)
- [A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy](https://mlsanity.com/paper/2302.10637/) (2023)
- [A Systematic Literature Review on Hardware Reliability Assessment Methods for Deep Neural Networks](https://mlsanity.com/paper/2305.05750/) (2023)
- [DVS-Attacks: Adversarial Attacks on Dynamic Vision Sensors for Spiking Neural Networks](https://mlsanity.com/paper/2107.00415/) (2021)
- [ReSpawn: Energy-Efficient Fault-Tolerance for Spiking Neural Networks considering Unreliable Memories](https://mlsanity.com/paper/2108.10271/) (2021)
- [Towards Energy-Efficient and Secure Edge AI: A Cross-Layer Framework](https://mlsanity.com/paper/2109.09829/) (2021)
- [Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2407.06992/) (2024)
- [MLDemon: Deployment Monitoring for Machine Learning Systems](https://mlsanity.com/paper/2104.13621/) (2021)
- [FitAct: Error Resilient Deep Neural Networks via Fine-Grained Post-Trainable Activation Functions](https://mlsanity.com/paper/2112.13544/) (2021)
- [DeepAxe: A Framework for Exploration of Approximation and Reliability Trade-offs in DNN Accelerators](https://mlsanity.com/paper/2303.08226/) (2023)
- [R-SNN: An Analysis and Design Methodology for Robustifying Spiking Neural Networks against Adversarial Attacks through Noise Filters for Dynamic Vision Sensors](https://mlsanity.com/paper/2109.00533/) (2021)
- [fakeWeather: Adversarial Attacks for Deep Neural Networks Emulating Weather Conditions on the Camera Lens of Autonomous Systems](https://mlsanity.com/paper/2205.13807/) (2022)
- [Isotropic Representation Can Improve Dense Retrieval](https://mlsanity.com/paper/2209.00218/) (2022)
- [DeepVigor: Vulnerability Value Ranges and Factors for DNNs' Reliability Assessment](https://mlsanity.com/paper/2303.06931/) (2023)
- [R2F: A Remote Retraining Framework for AIoT Processors with Computing Errors](https://mlsanity.com/paper/2107.03096/) (2021)
- [Winograd Convolution: A Perspective from Fault Tolerance](https://mlsanity.com/paper/2202.08675/) (2022)
- [Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systems](https://mlsanity.com/paper/2204.09514/) (2022)
- [Enhancing Fault Resilience of QNNs by Selective Neuron Splitting](https://mlsanity.com/paper/2306.09973/) (2023)
- [Data-Free Hard-Label Robustness Stealing Attack](https://mlsanity.com/paper/2312.05924/) (2023)
- [Fault-Tolerant Deep Learning: A Hierarchical Perspective](https://mlsanity.com/paper/2204.01942/) (2022)
- [Special Session: Approximation and Fault Resiliency of DNN Accelerators](https://mlsanity.com/paper/2306.04645/) (2023)
- [Exploiting Vulnerabilities in Deep Neural Networks: Adversarial and Fault-Injection Attacks](https://mlsanity.com/paper/2105.03251/) (2021)
- [RoHNAS: A Neural Architecture Search Framework with Conjoint Optimization for Adversarial Robustness and Hardware Efficiency of Convolutional and Capsule Networks](https://mlsanity.com/paper/2210.05276/) (2022)
- [SSAP: A Shape-Sensitive Adversarial Patch for Comprehensive Disruption of Monocular Depth Estimation in Autonomous Navigation Applications](https://mlsanity.com/paper/2403.11515/) (2024)
- [Bit Error Tolerance Metrics for Binarized Neural Networks](https://mlsanity.com/paper/2102.01344/) (2021)
- [Software Engineering Approaches for TinyML based IoT Embedded Vision: A Systematic Literature Review](https://mlsanity.com/paper/2204.08702/) (2022)
- [Statistical Modeling of Soft Error Influence on Neural Networks](https://mlsanity.com/paper/2210.05876/) (2022)
- [Implicit Bilevel Optimization: Differentiating through Bilevel Optimization Programming](https://mlsanity.com/paper/2302.14473/) (2023)
- [Probabilistic Safety Regions Via Finite Families of Scalable Classifiers](https://mlsanity.com/paper/2309.04627/) (2023)

## Abstract (arXiv)

Machine Learning (ML) techniques have been rapidly adopted by smart Cyber-Physical Systems (CPS) and Internet-of-Things (IoT) due to their powerful decision-making capabilities. However, they are vulnerable to various security and reliability threats, at both hardware and software levels, that compromise their accuracy. These threats get aggravated in emerging edge ML devices that have stringent constraints in terms of resources (e.g., compute, memory, power/energy), and that therefore cannot employ costly security and reliability measures. Security, reliability, and vulnerability mitigation techniques span from network security measures to hardware protection, with an increased interest towards formal verification of trained ML models. This paper summarizes the prominent vulnerabilities of modern ML systems, highlights successful defenses and mitigation techniques against these vulnerabilities, both at the cloud (i.e., during the ML training phase) and edge (i.e., during the ML inference stage), discusses the implications of a resource-constrained design on the reliability and security of the system, identifies verification methodologies to ensure correct system behavior, and describes open research challenges for building secure and reliable ML systems at both the edge and the cloud.