mlsanityData to 7 Oct 2026

FedMIX-P: Mixing Local and Global Preconditioners for Federated Vision and Language Model Training

arXiv 2610.01515 ↗First posted 1 Oct 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Adaptive preconditioners accelerate model training, but heterogeneous client geometries can bias federated updates even when gradients are evaluated at the same model. Round-start synchronization alone cannot prevent this mismatch from reappearing during local training. We propose textttFedMIX-P, which mixes shared and local preconditioners at every local step, retaining local adaptation while reducing mean-squared operator mismatch by a factor of λ2. For smooth nonconvex objectives with stochastic gradients and partial participation, we establish an O(R-1/2) stationarity bound using suitable stepsizes and a horizon-dependent mixing weight, without requiring local preconditioners to converge to one another. A two-client counterexample shows that fixed positive mixing can preserve a nonstationary fixed point. The theory covers bounded linear symmetric positive-definite preconditioners. Experiments with SOAP, Sophia, and Muon variants across vision and language tasks show improvements over corresponding local optimizers, including accuracy gains of up to 19.47 percentage points and lower validation loss for 60M--350M language models. Full nonlinear and momentum-based updates require separate analysis.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

19 of 24 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  7. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  8. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  9. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  10. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  11. 2025FedMuon: Accelerating Federated Learning with Matrix Orthogonalizationcited by 15
  12. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
7 more
  1. 2025DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Normcited by 9
  2. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 7
  3. 2026FedNSAM:Consistency of Local and Global Flatness for Federated Learningcited by 7
  4. 2026FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learningcited by 7
  5. 2024Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithmcited by 5
  6. 2026Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Datacited by 4
  7. 2025FedPM: Federated Learning Using Second-order Optimization with Preconditioned Mixing of Local Parameterscited by 3

Built on by

Not cited yet.