Federated Automatic Differentiation
Not on the map. Why?
Abstract (arXiv)
Federated learning (FL) is a general framework for learning across an axis of group partitioned data (heterogeneous clients) while preserving data privacy, under the orchestration of a central server. FL methods often compute gradients of loss functions purely locally (ie. entirely at each client, or entirely at the server), typically using automatic differentiation (AD) techniques. We propose a federated automatic differentiation (FAD) framework that 1) enables computing derivatives of functions involving client and server computation as well as communication between them and 2) operates in a manner compatible with existing federated technology. In other words, FAD computes derivatives across communication boundaries. We show, in analogy with traditional AD, that FAD may be implemented using various accumulation modes, which introduce distinct computation-communication trade-offs and systems requirements. Further, we show that a broad class of federated computations is closed under these various modes of FAD, implying in particular that if the original computation can be implemented using privacy-preserving primitives, its derivative may be computed using only these same primitives. We then show how FAD can be used to create algorithms that dynamically learn components of the algorithm itself. In particular, we show that FedAvg-style algorithms can exhibit significantly improved performance by using FAD to adjust the server optimization step automatically, or by using FAD to learn weighting schemes for computing weighted averages across clients.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 69 of its 98 references that are in the map. First posted on arXiv 18 Jan 2023.
Builds on
69 of 98 references.
References · 69
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
- 2020Adaptive Federated Optimizationcited by 1,036
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
- 2015Automatic differentiation in machine learning: a surveycited by 846
- 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
38 more
- 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
- 2019Towards Federated Learning at Scale: System Designcited by 826
- 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
- 2016Learning to learn by gradient descent by gradient descentcited by 616
- 2018Federated Learning for Mobile Keyboard Predictioncited by 565
- 2018Meta-Learning with Latent Embedding Optimizationcited by 476
- 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
- 2019Robust Aggregation for Federated Learningcited by 314
- 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
- 2019Differentially Private Learning with Adaptive Clippingcited by 218
- 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
- 2016Learning to Optimizecited by 195
- 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
- 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
- 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 159
- 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
- 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
- 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
- 2019Federated Learning for Emoji Prediction in a Mobile Keyboardcited by 115
- 2017Learned Optimizers that Scale and Generalizecited by 109
- 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
- 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
- 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 89
- 2021Federated Evaluation and Tuning for On-Device Personalization: System Design & Applicationscited by 84
- 2017Neural Optimizer Search with Reinforcement Learningcited by 82
- 2020Federated Bayesian Optimization via Thompson Samplingcited by 77
- 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
- 2021On Large-Cohort Training for Federated Learningcited by 73
- 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
- 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
- 2021Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharingcited by 58
- 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
- 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
- 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
- 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
- 2019Federated Learning Of Out-Of-Vocabulary Wordscited by 51
- 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
- 2017Learning to Optimize Neural Netscited by 41
50 most cited of 69 All 69 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.