mlsanityData to 7 Oct 2026

Private PoEtry: Private In-Context Learning via Product of Experts

arXiv 2602.05012 ↗First posted 4 Feb 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) enables Large Language Models (LLMs) to adapt to new tasks with only a small set of examples at inference time, thereby avoiding task-specific fine-tuning. However, in-context examples may contain privacy-sensitive information that should not be revealed through model outputs. Existing differential privacy (DP) approaches to ICL are either computationally expensive or rely on heuristics with limited effectiveness, including context oversampling, synthetic data generation, or unnecessary thresholding. We reformulate private ICL through the lens of a Product-of-Experts model. This gives a theoretically grounded framework, and the algorithm can be trivially parallelized. We evaluate our method across five datasets in text classification, math, and vision-language. We find that our method improves accuracy by more than 30 percentage points on average compared to prior DP-ICL methods, while maintaining strong privacy guarantees.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 57 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

45 of 57 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
33 more
  1. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  2. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  3. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  4. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  5. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  6. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  7. 2013The Composition Theorem for Differential Privacycited by 290
  8. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 237
  9. 2024Many-Shot In-Context Learningcited by 203
  10. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 94
  11. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  12. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  13. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  14. 2020Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacycited by 61
  15. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 56
  16. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  17. 2024Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Datacited by 49
  18. 2014Differential Privacy: An Economic Method for Choosing Epsiloncited by 34
  19. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 34
  20. 2024On the Privacy Risk of In-context Learningcited by 31
  21. 2025MCPToolBench++: A Large Scale AI Agent Model Context Protocol MCP Tool Use Benchmarkcited by 30
  22. 2024How Private are DP-SGD Implementations?cited by 22
  23. 2024Differentially Private Next-Token Prediction of Large Language Modelscited by 21
  24. 2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 21
  25. 2024Context-Aware Membership Inference Attacks against Pre-trained Large Language Modelscited by 20
  26. 2025VaultGemma: A Differentially Private Gemma Modelcited by 14
  27. 2024Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisationcited by 11
  28. 2024DP-TabICL: In-Context Learning with Differentially Private Tabular Datacited by 11
  29. 2025Logits are All We Need to Adapt Closed Modelscited by 4
  30. 2025ContextLeak: Auditing Leakage in Private In-Context Learning Methodscited by 4
  31. 2023Self-Supervised Open-Ended Classification with Small Visual Language Modelscited by 2
  32. 2024Convex Approximation of Two-Layer ReLU Networks for Hidden State Differential Privacycited by 2
  33. 2025NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptationcited by 1

Built on by

Not cited yet.