mlsanityData to 14 Oct 2026

Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysis

arXiv 2609.39464 ↗First posted 30 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In federated averaging, local objectives can admit multiple optimal heads, making the aggregate depend on which heads clients return. We study this ambiguity in federated multivariate regression with private backbones and a shared linear head, using an unconstrained feature model (UFM) that treats training-sample features as free variables. We introduce a natural selection rule: each client returns the optimal head closest to the broadcast head. We show that global minimization with a vanishing proximal penalty on the head realizes this rule. When the clients' optimal Gram matrices and the initial shared Gram matrix are positive definite, the shared Gram matrix follows a closed recursion and converges to the unique Bures-Wasserstein barycenter of the clients' optimal Gram matrices. Even with this alignment, the limit generally differs from the centralized optimal Gram matrix. We decompose this gap into three positive-semidefinite terms arising from differences in client target means, covariance heterogeneity, and averaging the aligned heads. A correction based on a one-time exchange of target means and covariances recovers the centralized optimal Gram matrix in one round under exact local optimization and the same selection rule. We verify these results numerically in the UFM and test its predictions on five tabular and five image regression datasets using deep networks with feature regularization and long local training. In these experiments, ordinary training approaches the predicted barycenter, while a weak proximal penalty improves endpoint agreement and yields trajectories that closely follow the predicted Gram dynamics. The correction moves the final Gram matrices close to the centralized UFM prediction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

43 of 58 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2018Group Normalizationcited by 1,524
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  8. 2020Federated Learning with Matched Averagingcited by 564
  9. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  10. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  11. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  12. 2019Model Fusion via Optimal Transportcited by 219
31 more
  1. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  2. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  3. 2015Appearance-Based Gaze Estimation in the Wildcited by 153
  4. 2020Neural collapse with unconstrained featurescited by 142
  5. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  6. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 102
  7. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  8. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  9. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 69
  10. 2021Visual Representation Learning Does Not Generalize Strongly Within the Same Domaincited by 61
  11. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 56
  12. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  13. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 52
  14. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  15. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 46
  16. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  17. 2023Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Datacited by 38
  18. 2022Perturbation Analysis of Neural Collapsecited by 35
  19. 2022FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networkscited by 23
  20. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  21. 2023Neural Collapse in Multi-label Learning with Pick-all-label Losscited by 14
  22. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  23. 2024Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Datacited by 10
  24. 2019Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extensioncited by 7
  25. 2025Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Modelcited by 6
  26. 2024Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Featurescited by 5
  27. 2025Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regimecited by 4
  28. 2025Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Modelcited by 2
  29. 2024Effectiveness of Distributed Gradient Descent with Local Steps for Overparameterized Modelscited by 1
  30. 2025The Other Side of the Coin: Unveiling the Downsides of Model Aggregation in Federated Learning from a Layer-peeled Perspectivecited by 1
  31. 2026Distribution Alignment for One-Shot Federated Learning via Optimal Transportcited by 1

Built on by

Not cited yet.