# Federated Language Models Under Bandwidth Budgets: Distillation Rates and Conformal Coverage

arXiv: https://arxiv.org/abs/2605.09986 · HTML: https://mlsanity.com/paper/2605.09986/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 11 May 2026.

## Builds on (most cited first, 11 of 11)

- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [FedKD: Communication Efficient Federated Learning via Knowledge Distillation](https://mlsanity.com/paper/2108.13323/) (2021)
- [TRAQ: Trustworthy Retrieval Augmented Question Answering via Conformal Prediction](https://mlsanity.com/paper/2307.04642/) (2023)
- [A Distributed One-Step Estimator](https://mlsanity.com/paper/1511.01443/) (2015)
- [Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality](https://mlsanity.com/paper/2506.20978/) (2025)
- [Principled Context Engineering for RAG: Statistical Guarantees via Conformal Prediction](https://mlsanity.com/paper/2511.17908/) (2025)
- [Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning](https://mlsanity.com/paper/2507.10348/) (2025)
- [Efficient Federated Conformal Prediction with Group-Conditional Guarantee](https://mlsanity.com/paper/2603.14198/) (2026)
- [Federated Conditional Conformal Prediction via Generative Models](https://mlsanity.com/paper/2510.13297/) (2025)

## Built on by (most cited first, 3 of 3)

- [Certified Adaptive Refresh: Anytime-Valid Monitoring for Federated Conformal RAG](https://mlsanity.com/paper/2605.29139/) (2026)
- [Matching Rates and Optimal Allocation for Federated Probe-Logit Distillation under Heterogeneous Bandwidth Budgets](https://mlsanity.com/paper/2605.29642/) (2026)
- [Byzantine-Robust Federated RAG via Aligned Calibration and Fixed-Membership Conformal Prediction](https://mlsanity.com/paper/2609.33037/) (2026)

## Abstract (arXiv)

Hospitals, companies, and research consortia want language models that learn from, and answer questions over, data that must stay where it is held, over links with limited bandwidth, such as the satellite links of remote clinics and ships. Averaging the sites' model parameters makes each message grow with the model and requires one shared architecture; methods that exchange predictions avoid both limits, but, to our knowledge, none bounds what compressing those predictions costs. We introduce two methods that compress every message at a small and predictable cost. Federated Probe-Logit Distillation (FPLD) trains a shared language model: each site sends compressed logits on shared unlabeled probes, and a hub distills one model from their average. Federated Conformal Retrieval-Augmented Generation (FC-RAG) answers questions: each site scores candidate answers using its own documents and sends compressed scores, and the hub returns a conformal prediction set that contains the correct answer at a guaranteed rate. Both use one codec, which centers each message and rounds it after adding a random offset known to the receiver. For logits within a fixed clipping range, we bound the expected log-loss added by compression independently of vocabulary size; the bound depends only on the bits, the range, and the number of sites, falls as more sites take part, and lets the bit budget be set before deployment. We also prove that recalibrating after compression keeps the coverage guarantee, and that FPLD's training error bounds how far coverage can slip when a threshold is reused after a model update. In experiments with GPT-2 and Qwen2.5, 8 bits per number cause no detectable loss in the trained model or in coverage. FPLD also trains a shared model across different architectures, where parameter averaging does not apply; with one shared architecture, parameter averaging remains more accurate.