mlsanityData to 14 Oct 2026

A Split-and-Privatize Framework for Large Language Model Fine-Tuning

arXiv 2312.15603 ↗First posted 25 Dec 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Fine-tuning is a prominent technique to adapt a pre-trained language model to downstream scenarios. In parameter-efficient fine-tuning, only a small subset of modules are trained over the downstream datasets, while leaving the rest of the pre-trained model frozen to save computation resources. In recent years, a popular productization form arises as Model-as-a-Service (MaaS), in which vendors provide abundant pre-trained language models, server resources and core functions, and customers can fine-tune, deploy and invoke their customized model by accessing the one-stop MaaS with their own private dataset. In this paper, we identify the model and data privacy leakage risks in MaaS fine-tuning, and propose a Split-and-Privatize (SAP) framework, which manage to mitigate the privacy issues by adapting the existing split learning architecture. The proposed SAP framework is sufficiently investigated by experiments, and the results indicate that it can enhance the empirical privacy by 62% at the cost of 1% model performance degradation on the Stanford Sentiment Treebank dataset.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 25 Dec 2023.

Builds on

25 of 34 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  10. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  11. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 385
  12. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
13 more
  1. 2020Information Leakage in Embedding Modelscited by 203
  2. 2015Inverting Visual Representations with Convolutional Networkscited by 170
  3. 2019Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbationscited by 119
  4. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 116
  5. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 78
  6. 2020Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairnesscited by 67
  7. 2021Natural Language Understanding with Privacy-Preserving BERTcited by 55
  8. 2023Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Servicescited by 43
  9. 2020SplitNN-driven Vertical Partitioningcited by 42
  10. 2023Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Predictioncited by 38
  11. 2021PyVertical: A Vertical Federated Learning Framework for Multi-headed SplitNNcited by 35
  12. 2020Towards Differentially Private Text Representationscited by 29
  13. 2022PAC Privacy: Automatic Privacy Measurement and Control of Data Processingcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 2.

11 papers.

  1. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  2. 2024No Free Lunch Theorem for Privacy-Preserving LLM Inferencecited by 13
  3. 2024A Middle Path for On-Premises LLM Deployment: Preserving Privacy Without Sacrificing Model Confidentialitycited by 8
  4. 2025A Novel Hat-Shaped Device-Cloud Collaborative Inference Framework for Large Language Modelscited by 5
  5. 2025VFLAIR-LLM: A Comprehensive Framework and Benchmark for Split Learning of LLMscited by 5
  6. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  7. 2024PPFedIT: Towards Privacy-Preserving Federated Instruction Tuning with Few-shot Local Examplescited by 3
  8. 2025Learning Obfuscations Of LLM Embedding Sequences: Stained Glass Transformcited by 2
  9. 2026From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Modelscited by 2
  10. 2026The Art of Mixology: Mixup-based Obfuscation for Privacy-Preserving Split Learning in Large Language Modelscited by 1
  11. 2025DistilLock: Safeguarding LLMs from Unauthorized Knowledge Distillation on the Edgecited by 0