mlsanityData to 14 Oct 2026

TallyTrain: Communication-Efficient Federated Distillation

arXiv 2607.00173 ↗First posted 30 Jun 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning is bandwidth-bound on two orthogonal axes: model size, which limits how often parameter-averaging methods can afford to merge, and class count, which makes per-probe soft-label distillation prohibitive at large vocabularies. Both ceilings tighten as modern systems scale. We collapse the class-count axis to lceil log2 C rceil bits per probe by transmitting only each peer's argmax class index, where C is the number of output classes. The resulting protocol, TallyTrain, is not merely compressed: under non-IID training it can be preferable to soft-label distillation, because under-trained peers are confidently wrong and majority voting filters this noise where soft-label averaging amplifies it. Across standard benchmarks, TallyTrain matches or beats soft-label distillation at up to three orders of magnitude less communication. We also relax the model-size axis: we compose the cheap hard-label consensus with sparse parameter merges to obtain a bandwidth-bridge variant, which Pareto-dominates every tested operating point of the standard FedAvg, FedProx and FedDF baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 29 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

24 of 29 references.

References · 24

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  8. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  9. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  11. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  12. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
12 more
  1. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  2. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  3. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  4. 2024Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchorscited by 21
  5. 2024Decentralized Sporadic Federated Learning: A Unified Algorithmic Framework with Convergence Guaranteescited by 10
  6. 2024FedSPD: A Soft-clustering Approach for Personalized Decentralized Federated Learningcited by 5
  7. 2024DFML: Decentralized Federated Mutual Learningcited by 4
  8. 2024NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernelcited by 4
  9. 2025Mitigating Privacy-Utility Trade-off in Decentralized Federated Learning via f-Differential Privacycited by 3
  10. 2023Competitive Advantage Attacks to Decentralized Federated Learningcited by 2
  11. 2025On the Fast Adaptation of Delayed Clients in Decentralized Federated Learning: A Centroid-Aligned Distillation Approachcited by 1
  12. 2026Decentralized Federated Learning by Partial Message Exchangecited by 1

Built on by

Not cited yet.