mlsanityData to 14 Oct 2026

CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learning

arXiv 2210.02912 ↗First posted 6 Oct 2022cs.LG cs.CRcited by 24

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is a setting for training machine learning models in distributed environments where the clients do not share their raw data but instead send model updates to a server. However, model updates can be subject to attacks and leak private information. Differential Privacy (DP) is a leading mitigation strategy which involves adding noise to clipped model updates, trading off performance for strong theoretical privacy guarantees. Previous work has shown that the threat model of DP is conservative and that the obtained guarantees may be vacuous or may overestimate information leakage in practice. In this paper, we aim to achieve a tighter measurement of the model exposure by considering a realistic threat model. We propose a novel method, CANIFE, that uses canaries - carefully crafted samples by a strong adversary to evaluate the empirical privacy of a training round. We apply this attack to vision models trained on CIFAR-10 and CelebA and to language models trained on Sent140 and Shakespeare. In particular, in realistic FL scenarios, we demonstrate that the empirical per-round epsilon obtained with CANIFE is 4-5x lower than the theoretical bound.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 46 of its 62 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

46 of 62 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  8. 2016Deep Learning with Differential Privacycited by 2,728
  9. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  10. 2020Extracting Training Data from Large Language Modelscited by 2,189
  11. 2020Adaptive Federated Optimizationcited by 1,046
  12. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
34 more
  1. 2018LEAF: A Benchmark for Federated Settingscited by 792
  2. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  3. 2017Evasion Attacks against Machine Learning at Test Timecited by 689
  4. 2017Renyi Differential Privacycited by 650
  5. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  6. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  7. 2013The Composition Theorem for Differential Privacycited by 297
  8. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 271
  9. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 236
  10. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  11. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 218
  12. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 204
  13. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 199
  14. 2021Numerical Composition of Differential Privacycited by 180
  15. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  16. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  17. 2022Reconstructing Training Data with Informed Adversariescited by 148
  18. 2021When the Curious Abandon Honesty: Federated Learning Is Not Privatecited by 127
  19. 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 108
  20. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 107
  21. 2021Gradient Inversion with Generative Image Priorcited by 101
  22. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 90
  23. 2019Hypothesis Testing Interpretations and Renyi Differential Privacycited by 85
  24. 2022Fishing for User Data in Large-Batch Federated Learning via Gradient Magnificationcited by 73
  25. 2022Recovering Private Text in Federated Learning of Language Modelscited by 70
  26. 2020Training Production Language Models without Memorizing User Datacited by 68
  27. 2019MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Samplingcited by 67
  28. 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 53
  29. 2020Understanding Unintended Memorization in Federated Learningcited by 36
  30. 2022Defending against Reconstruction Attacks with R'enyi Differential Privacycited by 32
  31. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  32. 2022Canary Extraction in Natural Language Understanding Modelscited by 21
  33. 2022Optimal Membership Inference Bounds for Adaptive Composition of Sampled Gaussian Mechanismscited by 19
  34. 2022Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devicescited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 8; 2025: 4; 2026: 6.

24 papers.

Sort:

By area:

  1. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  2. 2023Privacy Auditing with One (1) Training Runcited by 123
  3. 2023One-shot Empirical Privacy Estimation for Federated Learningcited by 36
  4. 2023Gaussian Membership Inference Privacycited by 21
  5. 2024Nearly Tight Black-Box Auditing of Differentially Private Machine Learningcited by 21
  6. 2023Unleashing the Power of Randomization in Auditing Differentially Private MLcited by 20
  7. 2024Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Modelcited by 18
  8. 2024The Elusive Pursuit of Reproducing PATE-GAN: Benchmarking, Auditing, Debuggingcited by 13
  9. 2024Auditing Differential Privacy Guarantees Using Density Estimationcited by 6
  10. 2024To Shuffle or not to Shuffle: Auditing DP-SGD with Shufflingcited by 6
  11. 2025Optimizing Canaries for Privacy Auditing with Metagradient Descentcited by 6
  12. 2025The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditingcited by 5
12 more
  1. 2024Analyzing Inference Privacy Risks Through Gradients in Machine Learningcited by 4
  2. 2024Some Targets Are Harder to Identify than Others: Quantifying the Target-dependent Membership Leakagecited by 3
  3. 2023Epsilon*: Privacy Metric for Machine Learning Modelscited by 2
  4. 2025On the Security and Privacy of Federated Learning: A Survey with Attacks, Defenses, Frameworks, Applications, and Future Directionscited by 2
  5. 2025MCMC for Bayesian estimation of Differential Privacy from Membership Inference Attackscited by 1
  6. 2026Sequential Membership Inference Attackscited by 1
  7. 2024Adversarial Sample-Based Approach for Tighter Privacy Auditing in Final Model-Only Scenarioscited by 0
  8. 2026Beyond Theoretical Bounds: Empirical Privacy Loss Calibration for Text Rewriting Under Local Differential Privacycited by 0
  9. 2026Detectability in Diversity: Improved Canary Crafting for Privacy Auditing in One Runcited by 0
  10. 2026Advancing the State-of-the-Art in Empirical Privacy Auditingcited by 0
  11. 2026Let's Ask Gauss: Improved One-Run Privacy Auditingcited by 0
  12. 2026Canaries in the Bank: Auditing User-Level Privacy in Private Evolutioncited by 0