# Neurotoxin: Durable Backdoors in Federated Learning

arXiv: https://arxiv.org/abs/2206.10341 · HTML: https://mlsanity.com/paper/2206.10341/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 12 Jun 2022.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates](https://mlsanity.com/paper/1803.01498/) (2018)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Sparsified SGD with Memory](https://mlsanity.com/paper/1809.07599/) (2018)
- [Analyzing Federated Learning through an Adversarial Lens](https://mlsanity.com/paper/1811.12470/) (2018)
- [The Hidden Vulnerability of Distributed Learning in Byzantium](https://mlsanity.com/paper/1802.07927/) (2018)
- [Attack of the Tails: Yes, You Really Can Backdoor Federated Learning](https://mlsanity.com/paper/2007.05084/) (2020)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [A Little Is Enough: Circumventing Defenses For Distributed Learning](https://mlsanity.com/paper/1902.06156/) (2019)
- [Can You Really Backdoor Federated Learning?](https://mlsanity.com/paper/1911.07963/) (2019)
- [Auditing Differentially Private Machine Learning: How Private is Private SGD?](https://mlsanity.com/paper/2006.07709/) (2020)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [Applied Federated Learning: Improving Google Keyboard Query Suggestions](https://mlsanity.com/paper/1812.02903/) (2018)
- [Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses](https://mlsanity.com/paper/2012.10544/) (2020)
- [FetchSGD: Communication-Efficient Federated Learning with Sketching](https://mlsanity.com/paper/2007.07682/) (2020)
- [Overcoming Forgetting in Federated Learning on Non-IID Data](https://mlsanity.com/paper/1910.07796/) (2019)
- [Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning](https://mlsanity.com/paper/2108.10241/) (2021)
- [CRFL: Certifiably Robust Federated Learning against Backdoor Attacks](https://mlsanity.com/paper/2106.08283/) (2021)
- [Learning to Detect Malicious Clients for Robust Federated Learning](https://mlsanity.com/paper/2002.00211/) (2020)

## Built on by (most cited first, 30 of 98)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark](https://mlsanity.com/paper/2311.06750/) (2023)
- [Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis](https://mlsanity.com/paper/2211.02408/) (2022)
- [Recent Advances on Federated Learning: A Systematic Survey](https://mlsanity.com/paper/2301.01299/) (2023)
- [Backdoor Attacks and Defenses in Federated Learning: Survey, Challenges and Future Research Directions](https://mlsanity.com/paper/2303.02213/) (2023)
- [Teach LLMs to Phish: Stealing Private Information from Language Models](https://mlsanity.com/paper/2403.00871/) (2024)
- [On the Vulnerability of Backdoor Defenses for Federated Learning](https://mlsanity.com/paper/2301.08170/) (2023)
- [FreqFed: A Frequency Analysis-Based Approach for Mitigating Poisoning Attacks in Federated Learning](https://mlsanity.com/paper/2312.04432/) (2023)
- [Chameleon: Adapting to Peer Images for Planting Durable Backdoors in Federated Learning](https://mlsanity.com/paper/2304.12961/) (2023)
- [FedSecurity: Benchmarking Attacks and Defenses in Federated Learning and Federated LLMs](https://mlsanity.com/paper/2306.04959/) (2023)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning](https://mlsanity.com/paper/2407.19845/) (2024)
- [Backdoor Federated Learning by Poisoning Backdoor-Critical Layers](https://mlsanity.com/paper/2308.04466/) (2023)
- [Data and Model Poisoning Backdoor Attacks on Wireless Federated Learning, and the Defense Mechanisms: A Comprehensive Survey](https://mlsanity.com/paper/2312.08667/) (2023)
- [Revisiting Personalized Federated Learning: Robustness Against Backdoor Attacks](https://mlsanity.com/paper/2302.01677/) (2023)
- [Do We Really Need to Design New Byzantine-robust Aggregation Rules?](https://mlsanity.com/paper/2501.17381/) (2025)
- [Get Rid Of Your Trail: Remotely Erasing Backdoors in Federated Learning](https://mlsanity.com/paper/2304.10638/) (2023)
- [Detecting Backdoor Attacks in Federated Learning via Direction Alignment Inspection](https://mlsanity.com/paper/2503.07978/) (2025)
- [Mitigating Memorization In Language Models](https://mlsanity.com/paper/2410.02159/) (2024)
- [BadVFL: Backdoor Attacks in Vertical Federated Learning](https://mlsanity.com/paper/2304.08847/) (2023)
- [G2uardFL: Safeguarding Federated Learning Against Backdoor Attacks through Attributed Client Graph Clustering](https://mlsanity.com/paper/2306.04984/) (2023)
- [A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspective](https://mlsanity.com/paper/2311.16065/) (2023)
- [ScionFL: Efficient and Robust Secure Quantized Aggregation](https://mlsanity.com/paper/2210.07376/) (2022)
- [FLTracer: Accurate Poisoning Attack Provenance in Federated Learning](https://mlsanity.com/paper/2310.13424/) (2023)
- [Practical and General Backdoor Attacks against Vertical Federated Learning](https://mlsanity.com/paper/2306.10746/) (2023)
- [Lurking in the shadows: Unveiling Stealthy Backdoor Attacks against Personalized Federated Learning](https://mlsanity.com/paper/2406.06207/) (2024)
- [Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Models](https://mlsanity.com/paper/2308.09490/) (2023)
- [ACE: A Model Poisoning Attack on Contribution Evaluation Methods in Federated Learning](https://mlsanity.com/paper/2405.20975/) (2024)
- [Adversarially Guided Stateful Defense Against Backdoor Attacks in Federated Deep Learning](https://mlsanity.com/paper/2410.11205/) (2024)
- [You Can Backdoor Personalized Federated Learning](https://mlsanity.com/paper/2307.15971/) (2023)

## Abstract (arXiv)

Due to their decentralized nature, federated learning (FL) systems have an inherent vulnerability during their training to adversarial backdoor attacks. In this type of attack, the goal of the attacker is to use poisoned updates to implant so-called backdoors into the learned model such that, at test time, the model's outputs can be fixed to a given target for certain inputs. (As a simple toy example, if a user types "people from New York" into a mobile keyboard app that uses a backdoored next word prediction model, then the model could autocomplete the sentence to "people from New York are rude"). Prior work has shown that backdoors can be inserted into FL models, but these backdoors are often not durable, i.e., they do not remain in the model after the attacker stops uploading poisoned updates. Thus, since training typically continues progressively in production FL systems, an inserted backdoor may not survive until deployment. Here, we propose Neurotoxin, a simple one-line modification to existing backdoor attacks that acts by attacking parameters that are changed less in magnitude during training. We conduct an exhaustive evaluation across ten natural language processing and computer vision tasks, and we find that we can double the durability of state of the art backdoors.