mlsanityData to 7 Oct 2026

"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice

arXiv 2212.14315 ↗First posted 29 Dec 2022cs.CR cs.LGcited by 55

Not on the map. Why?

Abstract (arXiv)

Recent years have seen a proliferation of research on adversarial machine learning. Numerous papers demonstrate powerful algorithmic attacks against a wide variety of machine learning (ML) models, and numerous other papers propose defenses that can withstand most attacks. However, abundant real-world evidence suggests that actual attackers use simple tactics to subvert ML-driven systems, and as a result security practitioners have not prioritized adversarial ML defenses. Motivated by the apparent gap between researchers and practitioners, this position paper aims to bridge the two domains. We first present three real-world case studies from which we can glean practical insights unknown or neglected in research. Next we analyze all adversarial ML papers recently published in top security conferences, highlighting positive trends and blind spots. Finally, we state positions on precise and cost-driven threat modeling, collaboration between industry and academia, and reproducible research. We believe that our positions, if adopted, will increase the real-world impact of future endeavours in adversarial ML, bringing both researchers and practitioners closer to their shared goal of improving the security of ML systems.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 92 of its 206 references that are in the map. First posted on arXiv 29 Dec 2022.

Builds on

92 of 206 references.

References · 92

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  7. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  8. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  9. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  10. 2019Adversarial Training for Free!cited by 638
  11. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  12. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
38 more
  1. 2019On Evaluating Adversarial Robustnesscited by 442
  2. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 411
  3. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  4. 2020FLTrust: Byzantine-robust Federated Learning via Trust Bootstrappingcited by 330
  5. 2019HopSkipJumpAttack: A Query-Efficient Decision-Based Attackcited by 328
  6. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  7. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  8. 2019High Accuracy and High Fidelity Extraction of Neural Networkscited by 208
  9. 2020Information Leakage in Embedding Modelscited by 199
  10. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  11. 2019Detecting AI Trojans Using Meta Neural Analysiscited by 192
  12. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 183
  13. 2020When Machine Unlearning Jeopardizes Privacycited by 181
  14. 2020Blind Backdoors in Deep Learning Modelscited by 174
  15. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 168
  16. 2020Membership Leakage in Label-Only Exposurescited by 156
  17. 2020Entangled Watermarks as a Defense against Model Extractioncited by 144
  18. 2019Privacy Risks of Securing Machine Learning Models against Adversarial Examplescited by 136
  19. 2019Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detectioncited by 121
  20. 2020Adversarial Machine Learning -- Industry Perspectivescited by 121
  21. 2020Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hidingcited by 121
  22. 2019Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learningcited by 117
  23. 2020Stealing Links from Graph Neural Networkscited by 114
  24. 2020Graph Backdoorcited by 112
  25. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 112
  26. 2014The Rise of Social Botscited by 96
  27. 2020Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Databasecited by 95
  28. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 89
  29. 2018Exploring Connections Between Active Learning and Model Extractioncited by 84
  30. 2018Interpretable Deep Learning under Firecited by 74
  31. 2021Practical Blind Membership Inference Attack via Differential Comparisonscited by 73
  32. 2019Who is Real Bob? Adversarial Attacks on Speaker Recognition Systemscited by 66
  33. 2020Dirty Road Can Attack: Security of Deep Learning based Automated Lane Centering under Physical-World Attackcited by 64
  34. 2018Motivating the Rules of the Game for Adversarial Example Researchcited by 57
  35. 2019Quantitative Verification of Neural Networks And its Security Applicationscited by 56
  36. 2021T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classificationcited by 56
  37. 2019Terminal Brain Damage: Exposing the Graceless Degradation in Deep Neural Networks Under Hardware Fault Attackscited by 55
  38. 2019Attacking Graph-based Classification via Manipulating the Graph Structurecited by 53

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 13; 2025: 17; 2026: 5.

55 papers.

Sort:

By area:

  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  3. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  4. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  5. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  6. 2023Provably safe systems: the only path to controllable AGIcited by 29
  7. 2024GPT-4 Jailbreaks Itself with Near-Perfect Success Using Self-Explanationcited by 24
  8. 2024Embedding-based classifiers can detect prompt injection attackscited by 24
  9. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  10. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 14
  11. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  12. 2023SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detectioncited by 13
38 more
  1. 2023Evading Black-box Classifiers Without Breaking Eggscited by 13
  2. 2023Efficient Query-Based Attack against ML-Based Android Malware Detection under Zero Knowledge Settingcited by 10
  3. 2025Token-Level Constraint Boundary Search for Jailbreaking Text-to-Image Modelscited by 9
  4. 2023Benchmarking Robustness to Adversarial Image Obfuscationscited by 8
  5. 2023MalProtect: Stateful Defense Against Adversarial Query Attacks in ML-based Malware Detectioncited by 8
  6. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attackscited by 8
  7. 2023Web Content Filtering through knowledge distillation of Large Language Modelscited by 7
  8. 2023PubDef: Defending Against Transfer Attacks From Public Modelscited by 7
  9. 2023SoK: Pitfalls in Evaluating Black-Box Attackscited by 6
  10. 2023Your Attack Is Too DUMB: Formalizing Attacker Scenarios for Adversarial Transferabilitycited by 5
  11. 2023Adversarial Machine Learning for Social Good: Reframing the Adversary as an Allycited by 5
  12. 2024SoK: On the Offensive Potential of AIcited by 5
  13. 2025Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigationcited by 5
  14. 2023The Adaptive Arms Race: Redefining Robustness in AI Securitycited by 4
  15. 2025On the Robustness of Transformers against Context Hijacking for Linear Classificationcited by 4
  16. 2026Adversarial News and Lost Profits: Manipulating Headlines in LLM-Driven Algorithmic Tradingcited by 4
  17. 2024Machine Learning in Space: Surveying the Robustness of on-board ML models to Radiationcited by 3
  18. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltrationcited by 3
  19. 2024Towards Non-Adversarial Algorithmic Recoursecited by 2
  20. 2025Defending against Adversarial Malware Attacks on ML-based Android Malware Detection Systemscited by 2
  21. 2025Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Datasetcited by 2
  22. 2025From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teamingcited by 2
  23. 2023Transferable Adversarial Robustness for Categorical Data via Universal Robust Embeddingscited by 1
  24. 2023Your Battery Is a Blast! Safeguarding Against Counterfeit Batteries with Authenticationcited by 1
  25. 2023Magmaw: Modality-Agnostic Adversarial Attacks on Machine Learning-Based Wireless Communication Systemscited by 1
  26. 2024Assessing biomedical knowledge robustness in large language models by query-efficient sampling attackscited by 1
  27. 2024SoK: Analyzing Adversarial Examples: A Framework to Study Adversary Knowledgecited by 1
  28. 2024A Novel Perturb-ability Score to Mitigate Evasion Adversarial Attacks on Flow-Based ML-NIDScited by 1
  29. 2024Taking off the Rose-Tinted Glasses: A Critical Look at Adversarial ML Through the Lens of Evasion Attackscited by 1
  30. 2025A Few Large Shifts: Layer-Inconsistency Based Minimal Overhead Adversarial Example Detectioncited by 1
  31. 2025Adversarial Machine Learning Attacks on Financial Reporting via Maximum Violated Multi-Objective Attackcited by 1
  32. 2025Automating AI Failure Tracking: Semantic Association of Reports in AI Incident Databasecited by 1
  33. 2023Adversarial Robustness on Image Classification with k-meanscited by 0
  34. 2024TA3: Testing Against Adversarial Attacks on Machine Learning Modelscited by 0
  35. 2025Expanding ML-Documentation Standards For Better Securitycited by 0
  36. 2025Certifiably robust malware detectors by designcited by 0
  37. 2025Surveying the Operational Cybersecurity and Supply Chain Threat Landscape when Developing and Deploying AI Systemscited by 0
  38. 2025Evaluating the Robustness of a Production Malware Detection System to Transferable Adversarial Attackscited by 0

50 most cited of 55 All 55 (JSON)