mlsanityData to 7 Oct 2026

Privacy-Preserving Instructions for Aligning Large Language Models

arXiv 2402.13659 ↗First posted 21 Feb 2024cs.CR cs.CLcited by 34

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Service providers of large language model (LLM) applications collect user instructions in the wild and use them in further aligning LLMs with users' intentions. These instructions, which potentially contain sensitive information, are annotated by human workers in the process. This poses a new privacy risk not addressed by the typical private optimization. To this end, we propose using synthetic instructions to replace real instructions in data annotation and model fine-tuning. Formal differential privacy is guaranteed by generating those synthetic instructions using privately fine-tuned generators. Crucial in achieving the desired utility is our novel filtering algorithm that matches the distribution of the synthetic instructions to that of the real ones. In both supervised fine-tuning and reinforcement learning from human feedback, our extensive experiments demonstrate the high utility of the final set of synthetic instructions by showing comparable results to real instructions. In supervised fine-tuning, models trained with private synthetic instructions outperform leading open-source models such as Vicuna.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 103 of its 135 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

103 of 135 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Privately Aligning Language Models with Reinforcement Learning25 Oct
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  3. 2020Learning to summarize from human feedback2 Sep

Other references · 100

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2016Deep Learning with Differential Privacycited by 2,682
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
38 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2017Billion-scale similarity search with GPUscited by 2,477
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  7. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  9. 2023Extracting Training Data from Diffusion Modelscited by 723
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  12. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  13. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  14. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  15. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  16. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  17. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  18. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  19. 2013The Composition Theorem for Differential Privacycited by 290
  20. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  21. 2021Numerical Composition of Differential Privacycited by 174
  22. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  23. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  24. 2020DP-CGAN: Differentially Private Synthetic Data and Label Generationcited by 118
  25. 2022Differentially Private Diffusion Modelscited by 118
  26. 2021Large-Scale Differentially Private BERTcited by 117
  27. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  28. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  29. 2021Winning the NIST Contest: A scalable and general approach to differentially private synthetic datacited by 101
  30. 2022The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymizationcited by 94
  31. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 94
  32. 2022AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Datacited by 88
  33. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  34. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  35. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  36. 2019Generative Models for Effective ML on Private, Decentralized Datasetscited by 85
  37. 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 85
  38. 2021Benchmarking Differentially Private Synthetic Data Generation Algorithmscited by 80

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 18; 2026: 6.

34 papers.

Sort:

By area:

  1. 2025Small Language Models are the Future of Agentic AIcited by 254
  2. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  3. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
  4. 2024Private prediction for large-scale synthetic text generationcited by 24
  5. 2024Synthesizing Post-Training Data for LLMs through Multi-Agent Simulationcited by 22
  6. 2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
  7. 2024Federated Learning in Practice: Reflections and Projectionscited by 15
  8. 2025Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMscited by 15
  9. 2025Can Textual Gradient Work in Federated Learning?cited by 14
  10. 2025POPri: Private Federated Learning using Preference-Optimized Synthetic Datacited by 14
  11. 2024Data-adaptive Differentially Private Prompt Synthesis for In-Context Learningcited by 10
  12. 2025MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Userscited by 10
22 more
  1. 2025Malicious LLM-Based Conversational AI Makes Users Reveal Personal Informationcited by 10
  2. 2025InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacycited by 10
  3. 2024LLM-based Privacy Data Augmentation Guided by Knowledge Distillation with a Distribution Tutor for Medical Text Classificationcited by 8
  4. 2025Cape: Context-Aware Prompt Perturbation Mechanism with Differential Privacycited by 5
  5. 2025Clustering and Median Aggregation Improve Differentially Private Inferencecited by 5
  6. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  7. 2024The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcarecited by 2
  8. 2025Synthesizing and Adapting Error Correction Data for Mobile Large Language Model Applicationscited by 2
  9. 2025ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Controlcited by 2
  10. 2026EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectorscited by 2
  11. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
  12. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  13. 2025PROPS: Progressively Private Self-alignment of Large Language Modelscited by 1
  14. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  15. 2026Differentially Private Preference Data Synthesis for Large Language Model Alignmentcited by 1
  16. 2025Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptationcited by 0
  17. 2025The End Of Universal Lifelong Identifiers: Identity Systems For The AI Eracited by 0
  18. 2025Secret-Protected Evolution for Differentially Private Synthetic Text Generationcited by 0
  19. 2026Feed m Birds with One Scone: Accelerating Multi-task Gradient Balancing via Bi-level Optimizationcited by 0
  20. 2026MAPLE: Metadata Augmented Private Language Evolutioncited by 0
  21. 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVRcited by 0
  22. 2026Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improved Algorithms for Differentially Private Language Model Alignment
  2. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data
  3. 2025Differentially Private Steering for Large Language Model Alignment
  4. 2026DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning
  5. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models
  6. 2026Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation
  7. 2025RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis
  8. 2024SafeSynthDP: Leveraging Large Language Models for Privacy-Preserving Synthetic Data Generation Using Differential Privacy