mlsanityData to 14 Oct 2026

DP-ES: Differentially Private Evolution Strategies for Prompt Optimization

arXiv 2610.06236 ↗First posted 5 Oct 2026cs.CR cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Token-level differentially private (DP) prompt optimization methods such as DP-OPT can become unstable under tight privacy budgets: on GSM8K, DP-OPT obtains 49.5±28.5% across 30 runs, and a logged search trajectory reveals prompt-template drift and noise-sensitive irreversible choices. We diagnose these as structural consequences of greedy token-by-token construction over privately aggregated counts. We then propose DP-ES (Differentially Private Evolution Strategies), a structurally cleaner alternative that maintains a population of full prompts, mutates them via LLM calls that never access the private dataset, and spends privacy only on sampled-Gaussian evaluation; deterministic or Gumbel-smoothed selection is post-processing. Under a conservative (varepsilon≤1.0,δ=10-5) guarantee, DP-ES achieves 88.1% on GSM8K (+38.6 pp over DP-OPT, approximately 9 times lower standard deviation), 99.7% on MedQA, 73.5% on BANKING77, and 86.8% on Alpaca. It is also 2.5 times faster in wall-clock time and uses 3.3 times fewer logged private-data call groups than DP-OPT. Selection and population ablations, implementation-level noise checks, and a 200-profile exact-match memorization stress test complement the formal guarantee. Scope: Our experiments establish optimization robustness under DP noise, especially where prompt structure is critical; end-to-end validation on genuinely sensitive, non-saturated deployment data remains future work.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

21 of 37 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2016Deep Learning with Differential Privacycited by 2,728
  5. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  7. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  9. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  10. 2018Scalable Private Learning with PATEcited by 269
  11. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 236
  12. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 218
9 more
  1. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 199
  2. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 98
  3. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  4. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 76
  5. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 65
  6. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 57
  7. 2024Privacy Preserving Prompt Engineering: A Surveycited by 25
  8. 2025SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigationcited by 19
  9. 2025Private LoRA Fine-tuning of Open-Source LLMs with Homomorphic Encryptioncited by 7

Built on by

Not cited yet.