mlsanityData to 7 Oct 2026

Differentially Private Attention Computation

arXiv 2305.04701 ↗First posted 8 May 2023cs.LG cs.CRcited by 25

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs), especially those based on the Transformer architecture, have had a profound impact on various aspects of daily life, such as natural language processing, content generation, research methodologies, and more. Nevertheless, a crucial concern regarding the inference results of large language models is the issue of security and privacy. Given that large language models can generate results that may leak sensitive confidential or copyright information in many scenarios, it is crucial to compute the attention matrix with provable privacy guarantees, as attention is all you need. In this work, we propose a novel and efficient algorithm for approximating the attention matrix while providing differential privacy (DP) guarantees. To achieve this, we build on recent advancements in fast attention computation and differentially private matrix publishing.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 36 of its 89 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

36 of 89 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2018YOLOv3: An Incremental Improvementcited by 2,652
  10. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  11. 2023A Watermark for Large Language Modelscited by 667
  12. 2021Differentially Private Fine-tuning of Language Modelscited by 392
24 more
  1. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  2. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 339
  3. 2017The Space of Transferable Adversarial Examplescited by 156
  4. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 136
  5. 2021Deep Learning with Label Differential Privacycited by 113
  6. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  7. 2023Fast Attention Requires Bounded Entriescited by 102
  8. 2018Solving Linear Programs in the Current Matrix Multiplication Timecited by 100
  9. 2023On Provable Copyright Protection for Generative Modelscited by 97
  10. 2023KDEformer: Accelerating Transformers via Kernel Density Estimationcited by 58
  11. 2019Solving Empirical Risk Minimization in the Current Matrix Multiplication Timecited by 53
  12. 2023Attention Scheme Inspired Softmax Regressioncited by 49
  13. 2023An Over-parameterized Exponential Regressioncited by 45
  14. 2020Faster Dynamic Matrix Inverse for Faster LPscited by 44
  15. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  16. 2019A Deterministic Linear Program Solver in Current Matrix Multiplication Timecited by 42
  17. 2022Privately Fine-Tuning Large Language Models with Differential Privacycited by 39
  18. 2023Algorithm and Hardness for Dynamic Attention Maintenance in Large Language Modelscited by 37
  19. 2023Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimensioncited by 37
  20. 2022A Faster Small Treewidth SDP Solvercited by 36
  21. 2023An Iterative Algorithm for Rescaled Hyperbolic Functions Regressioncited by 35
  22. 2020A Nearly-Linear Time Algorithm for Linear Programs with Small Treewidth: A Multiscale Representation of Robust Central Pathcited by 29
  23. 2022Sketching Meets Differential Privacy: Fast Algorithm for Dynamic Kronecker Projection Maintenancecited by 22
  24. 2023Differentially Private Continual Releases of Streaming Frequency Moment Estimationscited by 10

Built on by

Cited by year

Citing papers per year: 2023: 15; 2024: 8; 2025: 2.

25 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  3. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  4. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  5. 2024The Fine-Grained Complexity of Gradient Computation for Training Large Language Modelscited by 30
  6. 2023A Mathematical Abstraction for Balancing the Trade-off Between Creativity and Reality in Large Language Modelscited by 29
  7. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 29
  8. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  9. 2023GradientCoin: A Peer-to-Peer Decentralized Large Language Modelscited by 23
  10. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  11. 2023Solving Attention Kernel Regression Problem via Pre-conditionercited by 20
  12. 2023Beyond Classical Attention: Quantum Attention for Scalable Computationcited by 18
13 more
  1. 2023Zero-th Order Algorithm for Softmax Attention Optimizationcited by 16
  2. 2024Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyondcited by 15
  3. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  4. 2024Provable Differentially Private Computation of the Cross-Attention Mechanismcited by 14
  5. 2023A Unified Scheme of ResNet and Softmaxcited by 12
  6. 2023Toward Highly Efficient and Private Submodular Maximization via Matrix-Based Accelerationcited by 11
  7. 2023Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weightscited by 11
  8. 2025Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transformcited by 9
  9. 2023A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformercited by 6
  10. 2023Query Complexity of Active Learning for Function Family With Nearly Orthogonal Basiscited by 4
  11. 2025Subquadratic Algorithms and Hardness for Attention with Any Temperaturecited by 4
  12. 2024Differentially Private Kernel Density Estimationcited by 2
  13. 2024On Differentially Private String Distancescited by 0