# Implicit Federated In-context Learning For Task-Specific LLM Fine-Tuning

arXiv: https://arxiv.org/abs/2511.06757 · HTML: https://mlsanity.com/paper/2511.06757/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [The Learnability of In-Context Learning](https://mlsanity.com/paper/2303.07895/) (2023)
- [Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization](https://mlsanity.com/paper/2310.15080/) (2023)
- [Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning](https://mlsanity.com/paper/2406.15334/) (2024)
- [FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition](https://mlsanity.com/paper/2404.18848/) (2024)
- [FedAdapter: Efficient Federated Learning for Modern NLP](https://mlsanity.com/paper/2205.10162/) (2022)
- [FedPFT: Federated Proxy Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2404.11536/) (2024)
- [Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures](https://mlsanity.com/paper/2411.19128/) (2024)
- [Federated In-Context Learning: Iterative Refinement for Improved Answer Quality](https://mlsanity.com/paper/2506.07440/) (2025)
- [Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization](https://mlsanity.com/paper/2308.03275/) (2023)

## Abstract (arXiv)

As large language models continue to develop and expand, the extensive public data they rely on faces the risk of depletion. Consequently, leveraging private data within organizations to enhance the performance of large models has emerged as a key challenge. The federated learning paradigm, combined with model fine-tuning techniques, effectively reduces the number of trainable parameters. However,the necessity to process high-dimensional feature spaces results in substantial overall computational overhead. To address this issue, we propose the Implicit Federated In-Context Learning (IFed-ICL) framework. IFed-ICL draws inspiration from federated learning to establish a novel distributed collaborative paradigm, by converting client local context examples into implicit vector representations, it enables distributed collaborative computation during the inference phase and injects model residual streams to enhance model performance. Experiments demonstrate that our proposed method achieves outstanding performance across multiple text classification tasks. Compared to traditional methods, IFed-ICL avoids the extensive parameter updates required by conventional fine-tuning methods while reducing data transmission and local computation at the client level in federated learning. This enables efficient distributed context learning using local private-domain data, significantly improving model performance on specific tasks.