FedSEA-LLaMA: A Secure, Efficient and Adaptive Federated Splitting Framework for Large Language Models
Not on the map. Why?
Abstract (arXiv)
Private data holds promise for improving LLMs due to its high quality, but its scattered distribution across data silos and the high computational demands of LLMs limit their deployment in federated environments. To address this, the transformer-based federated split models are proposed, which offload most model parameters to the server (or distributed clients) while retaining only a small portion on the client to ensure data privacy. Despite this design, they still face three challenges: 1) Peer-to-peer key encryption struggles to secure transmitted vectors effectively; 2) The auto-regressive nature of LLMs means that federated split learning can only train and infer sequentially, causing high communication overhead; 3) Fixed partition points lack adaptability to downstream tasks. In this paper, we introduce FedSEA-LLaMA, a Secure, Efficient, and Adaptive Federated splitting framework based on LLaMA2. First, we inject Gaussian noise into forward-pass hidden states to enable secure end-to-end vector transmission. Second, we employ attention-mask compression and KV cache collaboration to reduce communication costs, accelerating training and inference. Third, we allow users to dynamically adjust the partition points for input/output blocks based on specific task requirements. Experiments on natural language understanding, summarization, and conversational QA tasks show that FedSEA-LLaMA maintains performance comparable to centralized LLaMA2 and achieves up to 8x speedups in training and inference. Further analysis of privacy attacks and different partition points also demonstrates the effectiveness of FedSEA-LLaMA in security and adaptability.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 21 May 2025.
Builds on
32 of 49 references.
References · 32
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
- 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
- 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
20 more
- 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
- 2018CoQA: A Conversational Question Answering Challengecited by 683
- 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
- 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
- 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
- 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
- 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
- 2020Unleashing the Tiger: Inference Attacks on Split Learningcited by 105
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
- 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
- 2020Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarizationcited by 72
- 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
- 2021UnSplit: Data-Oblivious Model Inversion, Model Stealing, and Label Inference Attacks Against Split Learningcited by 52
- 2018Learning Private Neural Language Modeling with Attentive Aggregationcited by 49
- 2022End-to-end Spoken Conversational Question Answering: Task, Dataset and Modelcited by 37
- 2020Neural Compression and Filtering for Edge-assisted Real-time Object Detection in Challenged Networkscited by 28
- 2021Topic-Aware Contrastive Learning for Abstractive Dialogue Summarizationcited by 23
- 2018A Qualitative Comparison of CoQA, SQuAD 2.0 and QuACcited by 22
- 2020Pretraining Federated Text Models for Next Word Predictioncited by 22
- 2024Promoting Data and Model Privacy in Federated Learning through Quantized LoRAcited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
- 2026The Safeguard Worked. Is the LLM System Safer?1 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2024Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Security
- 2025Enhancing Model Privacy in Federated Learning with Random Masking and Quantization
- 2026SafeLM: Unified Privacy-Aware Optimization for Trustworthy Federated Large Language Models
- 2025Oblivionis: A Lightweight Learning and Unlearning Framework for Federated Large Language Models
- 2026Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approach
- 2025PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation
- 2026FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization
- 2026Differentially Private and Communication Efficient Large Language Model Split Inference via Stochastic Quantization and Soft Prompt