mlsanityData to 7 Oct 2026

Manipulating SGD with Data Ordering Attacks

arXiv 2104.09667 ↗First posted 19 Apr 2021cs.LG cs.AI cs.CRcited by 73

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Machine learning is vulnerable to a wide variety of attacks. It is now well understood that by changing the underlying data distribution, an adversary can poison the model trained with it or introduce backdoors. In this paper we present a novel class of training-time attacks that require no changes to the underlying dataset or model architecture, but instead only change the order in which data are supplied to the model. In particular, we find that the attacker can either prevent the model from learning, or poison it to learn behaviours specified by the attacker. Furthermore, we find that even a single adversarially-ordered epoch can be enough to slow down model learning, or even to reset all of the learning progress. Indeed, the attacks presented here are not specific to the model or dataset, but rather target the stochastic nature of modern learning procedures. We extensively evaluate our attacks on computer vision and natural language benchmarks to find that the adversary can disrupt model training and even introduce backdoors.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 21 of its 37 references that are in the map. First posted on arXiv 19 Apr 2021.

Builds on

21 of 37 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  8. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  9. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  10. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  11. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  12. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
9 more
  1. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
  2. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  3. 2018Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learningcited by 219
  4. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  5. 2020Dynamic Backdoor Attacks Against Machine Learning Modelscited by 147
  6. 2020Sponge Examples: Energy-Latency Attacks on Neural Networkscited by 108
  7. 2017Practical Coreset Constructions for Machine Learningcited by 75
  8. 2020An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Biascited by 45
  9. 2020Poisoned classifiers are not only backdoored, they are fundamentally brokencited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 15; 2023: 17; 2024: 12; 2025: 14; 2026: 8.

73 papers.

Sort:

By area:

  1. 2021On the Necessity of Auditable Algorithmic Definitions for Machine Unlearningcited by 164
  2. 2021When the Curious Abandon Honesty: Federated Learning Is Not Privatecited by 126
  3. 2023Machine Unlearning: Solutions and Challengescited by 85
  4. 2022Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protectioncited by 80
  5. 2024Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacycited by 78
  6. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  7. 2022Black-box Dataset Ownership Verification via Backdoor Watermarkingcited by 57
  8. 2021Poison Forensics: Traceback of Data Poisoning Attacks in Neural Networkscited by 37
  9. 2024Fast Exact Unlearning for In-Context Learning Data for LLMscited by 33
  10. 2023Label Poisoning is All You Needcited by 27
  11. 2023On The Impact of Machine Learning Randomness on Group Fairnesscited by 26
  12. 2022Indiscriminate Data Poisoning Attacks on Neural Networkscited by 24
38 more
  1. 2022Architectural Backdoors in Neural Networkscited by 24
  2. 2022Verifiable and Provably Secure Machine Unlearningcited by 24
  3. 2023Tools for Verifying Neural Models' Training Datacited by 21
  4. 2023Distilling Cognitive Backdoor Patterns within an Imagecited by 20
  5. 2024Verification of Machine Unlearning is Fragilecited by 20
  6. 2023Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspectivecited by 19
  7. 2022Discrete Key-Value Bottleneckcited by 18
  8. 2023The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictionscited by 17
  9. 2023Towards more Practical Threat Models in Artificial Intelligence Securitycited by 17
  10. 2023Reconstructing Individual Data Points in Federated Learning Hardened with Differential Privacy and Secure Aggregationcited by 16
  11. 2025Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systemscited by 16
  12. 2022Characterizing & Finding Good Data Orderings for Fast Convergence of Sequential Gradient Methodscited by 15
  13. 2022Jigsaw Puzzle: Selective Backdoor Attack to Subvert Malware Classifierscited by 14
  14. 2025Test-Time Backdoor Detection for Object Detection Modelscited by 14
  15. 2024Backdoor Secrets Unveiled: Identifying Backdoor Data with Optimized Scaled Prediction Consistencycited by 13
  16. 2022Proof-of-Learning is Currently More Broken Than You Thinkcited by 12
  17. 2023FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensemblingcited by 11
  18. 2023Gradients Look Alike: Sensitivity is Often Overestimated in DP-SGDcited by 11
  19. 2021Accumulative Poisoning Attacks on Real-time Datacited by 10
  20. 2022Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimizationcited by 10
  21. 2022Sampling without Replacement Leads to Faster Rates in Finite-Sum Minimax Optimizationcited by 10
  22. 2024Architectural Neural Backdoors from First Principlescited by 10
  23. 2021SoK: Machine Learning Governancecited by 9
  24. 2022ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networkscited by 8
  25. 2024Edge Unlearning is Not "on Edge"! An Adaptive Exact Unlearning System on Resource-Constrained Devicescited by 8
  26. 2021Disrupting Model Training with Adversarial Shortcutscited by 7
  27. 2025SafeSplit: A Novel Defense Against Client-Side Backdoor Attacks in Split Learning (Full Version)cited by 7
  28. 2023Machine Learning needs Better Randomness Standards: Randomised Smoothing and PRNG-based attackscited by 6
  29. 2026Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noisecited by 6
  30. 2022A Survey on Poisoning Attacks Against Supervised Machine Learningcited by 5
  31. 2022Augmentation Backdoorscited by 5
  32. 2025SEPS: A Separability Measure for Robust Unlearning in LLMscited by 5
  33. 2025Fresh in memory: Training-order recency is linearly encoded in language model activationscited by 5
  34. 2023Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineeringcited by 4
  35. 2024TruVRF: Towards Triple-Granularity Verification on Machine Unlearningcited by 4
  36. 2025Machine Learning Models Have a Supply Chain Problemcited by 4
  37. 2024BadSampler: Harnessing the Power of Catastrophic Forgetting to Poison Byzantine-robust Federated Learningcited by 3
  38. 2023Understanding CNN Hidden Neuron Activations Using Structured Background Knowledge and Deductive Reasoningcited by 2

50 most cited of 73 All 73 (JSON)