mlsanityData to 7 Oct 2026

NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation

arXiv 2504.18147 ↗First posted 25 Apr 2025cs.CR cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLM) are typically trained on vast amounts of data from various sources. Even when designed modularly (e.g., Mixture-of-Experts), LLMs can leak privacy on their sources. Conversely, training such models in isolation arguably prohibits generalization. To this end, we propose a framework, NoEsis, which builds upon the desired properties of modularity, privacy, and knowledge transfer. NoEsis integrates differential privacy with a hybrid two-staged parameter-efficient fine-tuning that combines domain-specific low-rank adapters, acting as experts, with common prompt tokens, acting as a knowledge-sharing backbone. Results from our evaluation on CodeXGLUE showcase that NoEsis can achieve provable privacy guarantees with tangible knowledge transfer across domains, and empirically show protection against Membership Inference Attacks. Finally, on code completion tasks, NoEsis bridges at least 77% of the accuracy gap between the non-shared and the non-private baseline.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 46 of its 65 references that are in the map. First posted on arXiv 25 Apr 2025.

Builds on

46 of 65 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2016Deep Learning with Differential Privacycited by 2,682
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
34 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  4. 2021Membership Inference Attacks From First Principlescited by 839
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  6. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  7. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  8. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  9. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  10. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  11. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  12. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  13. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  14. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  15. 2024Mixture of LoRA Expertscited by 179
  16. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  17. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  18. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  19. 2023Modular Deep Learningcited by 82
  20. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  21. 2022Mixed Differential Privacy in Computer Visioncited by 53
  22. 2020Differentially Private Language Models Benefit from Public Pre-trainingcited by 46
  23. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Modelscited by 38
  24. 2024Multimodal Instruction Tuning with Conditional Mixture of LoRAcited by 36
  25. 2014Differential Privacy: An Economic Method for Choosing Epsiloncited by 34
  26. 2022M^4I: Multi-modal Models Membership Inferencecited by 30
  27. 2024Context-Aware Membership Inference Attacks against Pre-trained Large Language Modelscited by 20
  28. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  29. 2024Do LLMs Know to Respect Copyright Notice?cited by 12
  30. 2024Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generationcited by 7
  31. 2019Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Modelscited by 6
  32. 2022MultiCoder: Multi-Programming-Lingual Pre-Training for Low-Resource Code Completioncited by 6
  33. 2024Worldwide Federated Training of Language Modelscited by 5
  34. 2024Differentially Private Training of Mixture of Experts Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Private PoEtry: Private In-Context Learning via Product of Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation
  2. 2024Privacy-Preserving Instructions for Aligning Large Language Models
  3. 2026LLMs as Linguistic Chameleons: Decoupling Semantics and Structure for Privacy-Preserving Communication
  4. 2025Prada: Black-Box LLM Adaptation with Private Data on Resource-Constrained Devices
  5. 2026Securing LLMs in the Wild: Privacy and Security Challenges at the Edge
  6. 2026LLM-Redactor: An Empirical Evaluation of Eight Techniques for Privacy-Preserving LLM Requests
  7. 2025SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
  8. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data