# Federated Non-negative Matrix Factorization for Short Texts Topic Modeling with Mutual Information

arXiv: https://arxiv.org/abs/2205.13300 · HTML: https://mlsanity.com/paper/2205.13300/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 40 references that are in the map.
- First posted on arXiv 26 May 2022.

## Builds on (most cited first, 12 of 12)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [A Mutual Information Maximization Perspective of Language Representation Learning](https://mlsanity.com/paper/1910.08350/) (2019)
- [FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders](https://mlsanity.com/paper/2103.06413/) (2021)
- [Students Need More Attention: BERT-based AttentionModel for Small Data with Application to AutomaticPatient Message Triage](https://mlsanity.com/paper/2006.11991/) (2020)
- [An Annotated Dataset of Stack Overflow Post Edits](https://mlsanity.com/paper/2004.08193/) (2020)

## Built on by (most cited first, 5 of 5)

- [A Privacy-Preserving Subgraph-Level Federated Graph Neural Network via Differential Privacy](https://mlsanity.com/paper/2206.03492/) (2022)
- [Federated Knowledge Graph Completion via Latent Embedding Sharing and Tensor Factorization](https://mlsanity.com/paper/2311.10341/) (2023)
- [Federated Neural Topic Models](https://mlsanity.com/paper/2212.02269/) (2022)
- [Federated Topic Model and Model Pruning Based on Variational Autoencoder](https://mlsanity.com/paper/2311.00314/) (2023)
- [Constrained Non-negative Matrix Factorization for Guided Topic Modeling of Minority Topics](https://mlsanity.com/paper/2505.16493/) (2025)

## Abstract (arXiv)

Non-negative matrix factorization (NMF) based topic modeling is widely used in natural language processing (NLP) to uncover hidden topics of short text documents. Usually, training a high-quality topic model requires large amount of textual data. In many real-world scenarios, customer textual data should be private and sensitive, precluding uploading to data centers. This paper proposes a Federated NMF (FedNMF) framework, which allows multiple clients to collaboratively train a high-quality NMF based topic model with locally stored data. However, standard federated learning will significantly undermine the performance of topic models in downstream tasks (e.g., text classification) when the data distribution over clients is heterogeneous. To alleviate this issue, we further propose FedNMF+MI, which simultaneously maximizes the mutual information (MI) between the count features of local texts and their topic weight vectors to mitigate the performance degradation. Experimental results show that our FedNMF+MI methods outperform Federated Latent Dirichlet Allocation (FedLDA) and the FedNMF without MI methods for short texts by a significant margin on both coherence score and classification F1 score.