mlsanityData to 14 Oct 2026

Matching Rates and Optimal Allocation for Federated Probe-Logit Distillation under Heterogeneous Bandwidth Budgets

arXiv 2605.29642 ↗First posted 28 May 2026stat.ML cs.IT cs.LGcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

In federated language modeling, K nodes each hold n samples but cannot pool data or exchange full-precision gradients or weights. We study the minimax rate at which a conditional distribution over V tokens can be estimated when each node may upload at most B bits per query in a public probe set. In federated probe-logit distillation (FPLD), each node transmits a scalar-quantized logit vector on the probe set, and an aggregator distills a global parametric student. Prior work (Dubey and Huo, 2026) establishes a high-probability KL rate O(d/(Kn) + ρ√V log V / m + K-1 · 2-2B/V) plus optimization slack, with the bandwidth term in its trace-sharpened form. Whether this bandwidth-term rate is tight, and how the upper bound generalizes to heterogeneous per-node bandwidths, are left open. We close both gaps. First, the dithered FPLD construction has a matching single-round lower bound Omega(K-1 · 2-2B/V) under non-degeneracy, pinning the bandwidth-axis rate at Theta(K-1 · 2-2B/V). T-round sequential refinement with nested/scaled residual quantizers achieves O(K-1 · 2-2TB/V); vanilla FPLD's T-independent bandwidth term is suboptimal for every T > 1. Second, we establish a heterogeneous-bandwidth upper bound for per-node budgets Bi, paired with a closed-form optimal allocation Bi^* = Btot/K + (V/2) log2(wi / barwg), a log-tilted water-filling rule that is the per-node analogue of reverse water-filling for distortion-rate optimization. A plug-in adaptive variant estimates the weights from a short warm-up phase and attains 1 + O(√log(K/δ)/(m T0)) relative suboptimality. Synthetic n-gram simulations confirm that empirical KL is bracketed by the upper and lower bounds and that the optimal allocation strictly dominates uniform and inverse-weighted baselines under heterogeneous clipping.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 6 of its 7 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

6 of 7 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning14 Jul

Other references · 5

  1. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  2. 2018Inference under Information Constraints I: Lower Bounds from Chi-Square Contractioncited by 31
  3. 2014On Communication Cost of Distributed Statistical Estimation and Dimensionalitycited by 24
  4. 2015A Distributed One-Step Estimatorcited by 11
  5. 2026Federated Language Models Under Bandwidth Budgets: Distillation Rates and Conformal Coveragecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Communication-Aware Knowledge Distillation for Federated LLM Fine-Tuning over Wireless Networks
  2. 2026FedSDR: Federated Self-Distillation with Rectification
  3. 2026FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devices
  4. 2024FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillation
  5. 2026Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation
  6. 2025FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning
  7. 2026Federated Inference for Heterogeneous LLM Communication and Collaboration
  8. 2026What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute