# Attention, Anomalies! Handling Attention Layers in Unsupervised Federated Outlier Detection

arXiv: https://arxiv.org/abs/2608.04753 · HTML: https://mlsanity.com/paper/2608.04753/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Builds on (most cited first, 21 of 21)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [Privacy and Robustness in Federated Learning: Attacks and Defenses](https://mlsanity.com/paper/2012.06337/) (2020)
- [Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation](https://mlsanity.com/paper/1810.04304/) (2018)
- [Multi-Center Federated Learning: Clients Clustering for Better Personalization](https://mlsanity.com/paper/2108.08647/) (2021)
- [Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing](https://mlsanity.com/paper/2007.09236/) (2020)
- [FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure](https://mlsanity.com/paper/2010.06870/) (2020)
- [Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints](https://mlsanity.com/paper/1311.2106/) (2013)
- [apricot: Submodular selection for data summarization in Python](https://mlsanity.com/paper/1906.03543/) (2019)
- [A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithms](https://mlsanity.com/paper/2204.09825/) (2022)
- [Personalized Federated Learning via Convex Clustering](https://mlsanity.com/paper/2202.00718/) (2022)
- [FedHybrid: A Hybrid Primal-Dual Algorithm Framework for Federated Optimization](https://mlsanity.com/paper/2106.01279/) (2021)
- [Tax Evasion Risk Management Using a Hybrid Unsupervised Outlier Detection Method](https://mlsanity.com/paper/2103.01033/) (2021)
- [FedAD-Bench: A Unified Benchmark for Federated Unsupervised Anomaly Detection in Tabular Data](https://mlsanity.com/paper/2408.04442/) (2024)
- [Deep Learning Anomaly Detection for Cellular IoT with Applications in Smart Logistics](https://mlsanity.com/paper/2102.08936/) (2021)

## Abstract (arXiv)

Attention layers are the backbone of today's most powerful and impactful models. Models with multi-million and billion parameters rely on contextual knowledge provided by attention layers. However, their use goes well beyond just being the core component of large language models. One particularly interesting application is in Memory Augmented Autoencoders (MemAE), specifically for unsupervised representation learning in outlier detection tasks. It was shown that attention helps these models be more effective in centralized learning scenarios. Our work aims to address the lack of specialized aggregation techniques in Federated Learning (FL) when it comes to MemAE models. In this paper we analyze the intricacies of the architecture behind Memory Augmented Autoencoders, and propose novel, guided approaches to effectively aggregate these models in federated scenarios. We demonstrate our approach on non-IID datasets and show that these novel aggregation schemes are more robust when dealing with numerous edge nodes in environments with unbalanced datasets, specifically for unsupervised anomaly detection scenarios. This approach improves the performance of even very shallow autoencoders, allowing them to be used in resource constrained environments.