mlsanityData to 14 Oct 2026

Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfer

arXiv 2312.05842 ↗First posted 10 Dec 2023cs.AI cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

While large language models (LLMs) are empowered with broad knowledge, their task-specific performance is often suboptimal. It necessitates fine-tuning LLMs with task-specific data, but such data may be inaccessible due to privacy concerns. In this paper, we propose a novel approach to enhance LLMs with smaller language models (SLMs) that are trained on clients using their private task-specific data. To enable mutual enhancement between LLMs and SLMs, we propose CrossLM, where the SLMs promote the LLM to generate task-specific high-quality data, and both the LLM and SLMs are enhanced with the generated data. We evaluate CrossLM using publicly accessible language models across a range of benchmark tasks. The results demonstrate that CrossLM significantly enhances the task-specific performance of SLMs on clients and the LLM on the cloud server simultaneously while preserving the LLM's generalization capability.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 10 Dec 2023.

Builds on

29 of 38 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
17 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  7. 2018Hierarchical Neural Story Generationcited by 1,193
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  9. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  10. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  11. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 204
  12. 2019Federated Learning for Emoji Prediction in a Mobile Keyboardcited by 118
  13. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  14. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 78
  15. 2021Federated Learning Meets Natural Language Processing: A Surveycited by 43
  16. 2020Pretraining Federated Text Models for Next Word Predictioncited by 22
  17. 2022Federated Learning of Large Models at the Edge via Principal Sub-Model Trainingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  2. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  3. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 14
  4. 2024FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injectioncited by 8
  5. 2024FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Modelscited by 5
  6. 2024FuseGen: PLM Fusion for Data-generation based Zero-shot Learningcited by 4
  7. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  8. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  9. 2025Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysiscited by 1
  10. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
  11. 2024Federated Learning with New Knowledge: Fundamentals, Advances, and Futurescited by 0
  12. 2025FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learningcited by 0
1 more
  1. 2026Federated Reasoning Distillation Framework with Model Learnability-Aware Data Allocationcited by 0