mlsanityData to 14 Oct 2026

DiVa: An Accelerator for Differentially Private Machine Learning

arXiv 2208.12392 ↗First posted 26 Aug 2022cs.AR cs.AI cs.CRcited by 4

Not on the map. Why?

Abstract (arXiv)

The widespread deployment of machine learning (ML) is raising serious concerns on protecting the privacy of users who contributed to the collection of training data. Differential privacy (DP) is rapidly gaining momentum in the industry as a practical standard for privacy protection. Despite DP's importance, however, little has been explored within the computer systems community regarding the implication of this emerging ML algorithm on system designs. In this work, we conduct a detailed workload characterization on a state-of-the-art differentially private ML training algorithm named DP-SGD. We uncover several unique properties of DP-SGD (e.g., its high memory capacity and computation requirements vs. non-private ML), root-causing its key bottlenecks. Based on our analysis, we propose an accelerator for differentially private ML named DiVa, which provides a significant improvement in compute utilization, leading to 2.6x higher energy-efficiency vs. conventional systolic arrays.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 35 of its 89 references that are in the map. First posted on arXiv 26 Aug 2022.

Builds on

35 of 89 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2016Deep Learning with Differential Privacycited by 2,728
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  10. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  11. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 575
  12. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
23 more
  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  2. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  3. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  4. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  5. 2014cuDNN: Efficient Primitives for Deep Learningcited by 245
  6. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  7. 2017SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networkscited by 135
  8. 2021Large-Scale Differentially Private BERTcited by 118
  9. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 107
  10. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  11. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  12. 2020SpArch: Efficient Architecture for Sparse Matrix Multiplicationcited by 49
  13. 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 47
  14. 2019TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learningcited by 43
  15. 2017Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networkscited by 37
  16. 2020ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learningcited by 35
  17. 2019PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Unitscited by 30
  18. 2020Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendationscited by 21
  19. 2022DarKnight: An Accelerated Framework for Privacy and Integrity Preserving Deep Learning Using Trusted Hardwarecited by 14
  20. 2020Tensor Casting: Co-Designing Algorithm-Architecture for Personalized Recommendation Trainingcited by 13
  21. 2019Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learningcited by 9
  22. 2017Dropping Activation Outputs with Localized First-layer Deep Network for Enhancing User Privacy and Data Securitycited by 8
  23. 2019NeuMMU: Architectural Support for Efficient Address Translations in Neural Processing Unitscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Debunking the CUDA Myth Towards GPU-based AI Systemscited by 2
  2. 2024LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Modelscited by 1
  3. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  4. 2025Empowering Vector Architectures for ML: The CAMP Architecture for Matrix Multiplicationcited by 0