# Adam in Private: Secure and Fast Training of Deep Neural Networks with Adaptive Moment Estimation

arXiv: https://arxiv.org/abs/2106.02203 · HTML: https://mlsanity.com/paper/2106.02203/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 69 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://mlsanity.com/paper/1902.09843/) (2019)
- [FALCON: Honest-Majority Maliciously Secure Framework for Private Deep Learning](https://mlsanity.com/paper/2004.02229/) (2020)
- [Chameleon: A Hybrid Secure Computation Framework for Machine Learning Applications](https://mlsanity.com/paper/1801.03239/) (2018)
- [CrypTFlow: Secure TensorFlow Inference](https://mlsanity.com/paper/1909.07814/) (2019)
- [XONN: XNOR-based Oblivious Deep Neural Network Inference](https://mlsanity.com/paper/1902.07342/) (2019)
- [DeepSecure: Scalable Provably-Secure Deep Learning](https://mlsanity.com/paper/1705.08963/) (2017)
- [CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU](https://mlsanity.com/paper/2104.10949/) (2021)
- [QUOTIENT: Two-Party Secure Neural Network Training and Prediction](https://mlsanity.com/paper/1907.03372/) (2019)
- [BLAZE: Blazing Fast Privacy-Preserving Machine Learning](https://mlsanity.com/paper/2005.09042/) (2020)
- [Glyph: Fast and Accurately Training Deep Neural Networks on Encrypted Data](https://mlsanity.com/paper/1911.07101/) (2019)
- [Trident: Efficient 4PC Framework for Privacy Preserving Machine Learning](https://mlsanity.com/paper/1912.02631/) (2019)
- [Secure Evaluation of Quantized Neural Networks](https://mlsanity.com/paper/1910.12435/) (2019)
- [SWIFT: Super-fast and Robust Privacy-Preserving Machine Learning](https://mlsanity.com/paper/2005.10296/) (2020)
- [ASTRA: High Throughput 3PC over Rings with Application to Secure Prediction](https://mlsanity.com/paper/1912.02592/) (2019)
- [BAYHENN: Combining Bayesian Deep Learning and Homomorphic Encryption for Secure DNN Inference](https://mlsanity.com/paper/1906.00639/) (2019)
- [PrivEdge: From Local to Distributed Private Training and Prediction](https://mlsanity.com/paper/2004.05574/) (2020)
- [Effectiveness of MPC-friendly Softmax Replacement](https://mlsanity.com/paper/2011.11202/) (2020)
- [MOBIUS: Model-Oblivious Binarized Neural Networks](https://mlsanity.com/paper/1811.12028/) (2018)

## Built on by (most cited first, 4 of 4)

- [CryptoTrain: Fast Secure Training on Encrypted Dataset](https://mlsanity.com/paper/2409.16675/) (2024)
- [Wildest Dreams: Reproducible Research in Privacy-preserving Neural Network Training](https://mlsanity.com/paper/2403.03592/) (2024)
- [Threats, Vulnerabilities, and Controls of Machine Learning Based Systems: A Survey and Taxonomy](https://mlsanity.com/paper/2301.07474/) (2023)
- [Enhanced Artificial Neural Networks Using QHAdamW in Air Quality Forecasting](https://mlsanity.com/paper/2608.21463/) (2026)

## Abstract (arXiv)

Privacy-preserving machine learning (PPML) aims at enabling machine learning (ML) algorithms to be used on sensitive data. We contribute to this line of research by proposing a framework that allows efficient and secure evaluation of full-fledged state-of-the-art ML algorithms via secure multi-party computation (MPC). This is in contrast to most prior works, which substitute ML algorithms with approximated "MPC-friendly" variants. A drawback of the latter approach is that fine-tuning of the combined ML and MPC algorithms is required, which might lead to less efficient algorithms or inferior quality ML. This is an issue for secure deep neural networks (DNN) training in particular, as this involves arithmetic algorithms thought to be "MPC-unfriendly", namely, integer division, exponentiation, inversion, and square root. In this work, we propose secure and efficient protocols for the above seemingly MPC-unfriendly computations. Our protocols are three-party protocols in the honest-majority setting, and we propose both passively secure and actively secure with abort variants. A notable feature of our protocols is that they simultaneously provide high accuracy and efficiency. This framework enables us to efficiently and securely compute modern ML algorithms such as Adam and the softmax function "as is", without resorting to approximations. As a result, we obtain secure DNN training that outperforms state-of-the-art three-party systems; our full training is up to 6.7 times faster than just the online phase of the recently proposed FALCON@PETS'21 on a standard benchmark network. We further perform measurements on real-world DNNs, AlexNet and VGG16. The performance of our framework is up to a factor of about 12-14 faster for AlexNet and 46-48 faster for VGG16 to achieve an accuracy of 70% and 75%, respectively, when compared to FALCON.