# Federated Inference for Heterogeneous LLM Communication and Collaboration

arXiv: https://arxiv.org/abs/2603.28772 · HTML: https://mlsanity.com/paper/2603.28772/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2026.

## Built on by (most cited first, 2 of 2)

- [KV Cache Translation across Heterogeneous Large Language Models](https://mlsanity.com/paper/2607.28979/) (2026)
- [From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivity](https://mlsanity.com/paper/2609.10714/) (2026)

## Abstract (arXiv)

Given the limited performance and efficiency of on-device Large Language Models (LLMs), the collaborations between multiple LLMs enable desirable performance enhancements, in which data, tokens, and model weights could be shared across LLMs. This process is constrained by task-oriented QoS demands, privacy requirements, and inherent system heterogeneity. In view of the above challenge and to fully exploit the on-device inference capabilities, we present a novel federated inference framework in this position paper, termed federated refinement textttFedRefine. This framework presents a new paradigm for heterogeneous LLMs collaboratively performing inference with communicating KV caches in a privacy-preserving manner. Some numerical results are provided to highlight the superiority of textttFedRefine. Several interesting topics are also highlighted for future research. By exploring the LLM-native communications, we wish to provide a new paradigm for this broad area.