# FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity

arXiv: https://arxiv.org/abs/2607.03763 · HTML: https://mlsanity.com/paper/2607.03763/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 4 Jul 2026.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning](https://mlsanity.com/paper/2008.03606/) (2020)
- [FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data](https://mlsanity.com/paper/2005.11418/) (2020)
- [Adam-mini: Use Fewer Learning Rates To Gain More](https://mlsanity.com/paper/2406.16793/) (2024)
- [OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning](https://mlsanity.com/paper/2402.06954/) (2024)
- [Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning](https://mlsanity.com/paper/2010.05627/) (2020)
- [Why Transformers Need Adam: A Hessian Perspective](https://mlsanity.com/paper/2402.16788/) (2024)
- [SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum](https://mlsanity.com/paper/1910.00643/) (2019)
- [FedCM: Federated Learning with Client-level Momentum](https://mlsanity.com/paper/2106.10874/) (2021)
- [Cautious Optimizers: Improving Training with One Line of Code](https://mlsanity.com/paper/2411.16085/) (2024)
- [FedADC: Accelerated Federated Learning with Drift Control](https://mlsanity.com/paper/2012.09102/) (2020)
- [Communication-Efficient Federated Learning with Accelerated Client Gradient](https://mlsanity.com/paper/2201.03172/) (2022)
- [Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup](https://mlsanity.com/paper/2308.00522/) (2023)
- [FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models](https://mlsanity.com/paper/2510.27486/) (2025)
- [On Surprising Effectiveness of Masking Updates in Adaptive Optimizers](https://mlsanity.com/paper/2602.15322/) (2026)
- [Gradient Correction in Federated Learning with Adaptive Optimization](https://mlsanity.com/paper/2502.02727/) (2025)
- [MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization](https://mlsanity.com/paper/2606.17526/) (2026)

## Abstract (arXiv)

Federated Transformer training increasingly relies on local AdamW, whose adaptive updates can provide much stronger local progress than SGD-based training. However, under heterogeneous client data, even globally corrected AdamW updates may remain highly uneven in coordinate-wise reliability. We refer to this phenomenon as coordinate trust mismatch. Existing federated adaptive optimizers mainly address mismatch at the client-update or communication-round level, but still apply the corrected adaptive direction densely and uniformly across coordinates. In this paper, we propose FedACT, a global-aware coordinate trust modulation method for federated AdamW training. FedACT first forms a globally corrected adaptive direction and then reallocates update magnitudes according to a coordinate-wise trust score, assigning larger steps to coordinates jointly supported by local gradients and global correction, while preserving smaller non-zero updates on the remaining coordinates. Extensive experiments on federated vision Transformers, CNNs, LLM pre-training, and LLM fine-tuning show that FedACT consistently improves over strong federated adaptive baselines, with the largest gains on Transformer models under stronger data heterogeneity. Mechanism analyses further show that FedACT improves cross-client direction consistency, suggesting that coordinate-level trust allocation effectively complements round-level global-local correction. Code will be released.