# Federated Learning with Position-Aware Neurons

arXiv: https://arxiv.org/abs/2203.14666 · HTML: https://mlsanity.com/paper/2203.14666/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 55 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Federated Machine Learning: Concept and Applications](https://mlsanity.com/paper/1902.04885/) (2019)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [FedML: A Research Library and Benchmark for Federated Machine Learning](https://mlsanity.com/paper/2007.13518/) (2020)
- [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [FedScale: Benchmarking Model and System Performance of Federated Learning at Scale](https://mlsanity.com/paper/2105.11367/) (2021)
- [Positional Encoding as Spatial Inductive Bias in GANs](https://mlsanity.com/paper/2012.05217/) (2020)
- [What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding](https://mlsanity.com/paper/2010.04903/) (2020)
- [Fed2: Feature-Aligned Federated Learning](https://mlsanity.com/paper/2111.14248/) (2021)

## Built on by (most cited first, 24 of 24)

- [Personalized Federated Learning via Variational Bayesian Inference](https://mlsanity.com/paper/2206.07977/) (2022)
- [Towards Efficient Replay in Federated Incremental Learning](https://mlsanity.com/paper/2403.05890/) (2024)
- [Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup](https://mlsanity.com/paper/2308.00522/) (2023)
- [A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency](https://mlsanity.com/paper/2307.10655/) (2023)
- [Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2210.06671/) (2022)
- [AFL: A Single-Round Analytic Approach for Federated Learning with Pre-trained Models](https://mlsanity.com/paper/2405.16240/) (2024)
- [FedUV: Uniformity and Variance for Heterogeneous Federated Learning](https://mlsanity.com/paper/2402.18372/) (2024)
- [Federated Learning with Manifold Regularization and Normalized Update Reaggregation](https://mlsanity.com/paper/2311.05924/) (2023)
- [FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning](https://mlsanity.com/paper/2311.13267/) (2023)
- [Improving Model Fusion by Training-time Neuron Alignment with Fixed Neuron Anchors](https://mlsanity.com/paper/2402.01342/) (2024)
- [MAP: Model Aggregation and Personalization in Federated Learning with Incomplete Classes](https://mlsanity.com/paper/2404.09232/) (2024)
- [Avoid Overfitting User Specific Information in Federated Keyword Spotting](https://mlsanity.com/paper/2206.08864/) (2022)
- [Federated Learning Client Pruning for Noisy Labels](https://mlsanity.com/paper/2411.07391/) (2024)
- [MrTF: Model Refinery for Transductive Federated Learning](https://mlsanity.com/paper/2305.04201/) (2023)
- [Weight Scope Alignment: A Frustratingly Easy Method for Model Merging](https://mlsanity.com/paper/2408.12237/) (2024)
- [FedGA: A Fair Federated Learning Framework Based on the Gini Coefficient](https://mlsanity.com/paper/2507.12983/) (2025)
- [Tensor Decomposition based Personalized Federated Learning](https://mlsanity.com/paper/2208.12959/) (2022)
- [Personalized Federated Learning via Gradient Modulation for Heterogeneous Text Summarization](https://mlsanity.com/paper/2304.11524/) (2023)
- [Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks](https://mlsanity.com/paper/2405.12489/) (2024)
- [Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks](https://mlsanity.com/paper/2405.12493/) (2024)
- [FedDyMem: Efficient Federated Learning with Dynamic Memory and Memory-Reduce for Unsupervised Image Anomaly Detection](https://mlsanity.com/paper/2502.21012/) (2025)
- [FedAgg: Adaptive Federated Learning with Aggregated Gradients](https://mlsanity.com/paper/2303.15799/) (2023)
- [Selective Knowledge Sharing for Personalized Federated Learning Under Capacity Heterogeneity](https://mlsanity.com/paper/2405.20589/) (2024)
- [UniVarFL: Uniformity and Variance Regularized Federated Learning for Heterogeneous Data](https://mlsanity.com/paper/2506.08167/) (2025)

## Abstract (arXiv)

Federated Learning (FL) fuses collaborative models from local nodes without centralizing users' data. The permutation invariance property of neural networks and the non-i.i.d. data across clients make the locally updated parameters imprecisely aligned, disabling the coordinate-based parameter averaging. Traditional neurons do not explicitly consider position information. Hence, we propose Position-Aware Neurons (PANs) as an alternative, fusing position-related values (i.e., position encodings) into neuron outputs. PANs couple themselves to their positions and minimize the possibility of dislocation, even updating on heterogeneous data. We turn on/off PANs to disable/enable the permutation invariance property of neural networks. PANs are tightly coupled with positions when applied to FL, making parameters across clients pre-aligned and facilitating coordinate-based parameter averaging. PANs are algorithm-agnostic and could universally improve existing FL algorithms. Furthermore, "FL with PANs" is simple to implement and computationally friendly.