mlsanityData to 7 Oct 2026

Towards a Robust and Trustworthy Machine Learning System Development: An Engineering Perspective

arXiv 2101.03042 ↗First posted 8 Jan 2021cs.LG cs.CRcited by 9

Not on the map. Why?

Abstract (arXiv)

While Machine Learning (ML) technologies are widely adopted in many mission critical fields to support intelligent decision-making, concerns remain about system resilience against ML-specific security attacks and privacy breaches as well as the trust that users have in these systems. In this article, we present our recent systematic and comprehensive survey on the state-of-the-art ML robustness and trustworthiness from a security engineering perspective, focusing on the problems in system threat analysis, design and evaluation faced in developing practical machine learning applications, in terms of robustness and user trust. Accordingly, we organize the presentation of this survey intended to facilitate the convey of the body of knowledge from this angle. We then describe a metamodel we created that represents the body of knowledge in a standard and visualized way. We further illustrate how to leverage the metamodel to guide a systematic threat analysis and security design process which extends and scales up the classic process. Finally, we propose the future research directions motivated by our findings. Our work differs itself from the existing surveys by (i) exploring the fundamental principles and best practices to support robust and trustworthy ML system development, and (ii) studying the interplay of robustness and user trust in the context of ML systems. We expect this survey provides a big picture for machine learning security practitioners.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 73 of its 171 references that are in the map. First posted on arXiv 8 Jan 2021.

Builds on

73 of 171 references.

References · 73

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  7. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  8. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  9. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  10. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  11. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  12. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
38 more
  1. 2016Universal adversarial perturbationscited by 774
  2. 2018How To Backdoor Federated Learningcited by 745
  3. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  4. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  5. 2017One pixel attack for fooling deep neural networkscited by 579
  6. 2017Countering Adversarial Images using Input Transformationscited by 517
  7. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  8. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 484
  9. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 479
  10. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
  11. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 462
  12. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
  13. 2019On Evaluating Adversarial Robustnesscited by 442
  14. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 411
  15. 2019Hidden Trigger Backdoor Attackscited by 359
  16. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 350
  17. 2019Label Propagation for Deep Semi-supervised Learningcited by 287
  18. 2018Scalable Private Learning with PATEcited by 266
  19. 2014RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Responsecited by 262
  20. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  21. 2016A study of the effect of JPG compression on adversarial imagescited by 196
  22. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 183
  23. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 163
  24. 2019Principles alone cannot guarantee ethical AIcited by 134
  25. 2018Stealing Hyperparameters in Machine Learningcited by 115
  26. 2020Intrinsic Certified Robustness of Bagging against Data Poisoning Attackscited by 90
  27. 2016Towards the Science of Security and Privacy in Machine Learningcited by 89
  28. 2017Keeping the Bad Guys Out: Protecting and Vaccinating Deep Learning with JPEG Compressioncited by 86
  29. 2018Privacy Preserving Machine Learning: Threats and Solutionscited by 78
  30. 2018Adversarial Examples: Opportunities and Challengescited by 62
  31. 2020Influence Function based Data Poisoning Attacks to Top-N Recommender Systemscited by 61
  32. 2018AttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine Learningcited by 58
  33. 2018Low Latency Privacy Preserving Inferencecited by 57
  34. 2019Certified Robustness for Top-k Predictions against Adversarial Perturbations via Randomized Smoothingcited by 54
  35. 2018Poisoning Attacks to Graph-Based Recommender Systemscited by 52
  36. 2017Defense against Universal Adversarial Perturbationscited by 48
  37. 2015A Unified Gradient Regularization Family for Adversarial Examplescited by 43
  38. 2016Adversarial Perturbations Against Deep Neural Networks for Malware Classificationcited by 42

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 3; 2024: 0; 2025: 2.

9 papers.

  1. 2023SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detectioncited by 13
  2. 2022Towards a multi-stakeholder value-based assessment framework for algorithmic systemscited by 12
  3. 2023fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Taskscited by 10
  4. 2022A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunitiescited by 8
  5. 2021Machine Learning for Security in Vehicular Networks: A Comprehensive Surveycited by 7
  6. 2025An Investigation of Visual Foundation Models Robustnesscited by 2
  7. 2023Overcoming Adversarial Attacks for Human-in-the-Loop Applicationscited by 1
  8. 2025Human-AI Interaction and User Satisfaction: Empirical Evidence from Online Reviews of AI Productscited by 1
  9. 2021An Interpretable Graph-based Mapping of Trustworthy Machine Learning Researchcited by 0