# Federated Topic Model and Model Pruning Based on Variational Autoencoder

arXiv: https://arxiv.org/abs/2311.00314 · HTML: https://mlsanity.com/paper/2311.00314/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 43 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Federated Optimization: Distributed Machine Learning for On-Device Intelligence](https://mlsanity.com/paper/1610.02527/) (2016)
- [Model Pruning Enables Efficient Federated Learning on Edge Devices](https://mlsanity.com/paper/1909.12326/) (2019)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder](https://mlsanity.com/paper/2205.12035/) (2022)
- [Filter-enhanced MLP is All You Need for Sequential Recommendation](https://mlsanity.com/paper/2202.13556/) (2022)
- [Self-Supervised Graph Co-Training for Session-based Recommendation](https://mlsanity.com/paper/2108.10560/) (2021)
- [Scientific and Technological Information Oriented Semantics-adversarial and Media-adversarial Cross-media Retrieval](https://mlsanity.com/paper/2203.08615/) (2022)
- [T2-GNN: Graph Neural Networks for Graphs with Incomplete Features and Structure via Teacher-Student Distillation](https://mlsanity.com/paper/2212.12738/) (2022)
- [Federated Non-negative Matrix Factorization for Short Texts Topic Modeling with Mutual Information](https://mlsanity.com/paper/2205.13300/) (2022)

## Abstract (arXiv)

Topic modeling has emerged as a valuable tool for discovering patterns and topics within large collections of documents. However, when cross-analysis involves multiple parties, data privacy becomes a critical concern. Federated topic modeling has been developed to address this issue, allowing multiple parties to jointly train models while protecting privacy. However, there are communication and performance challenges in the federated scenario. To solve these problems, this paper proposes a method to establish a federated topic model while ensuring the privacy of each node and uses neural network model pruning to accelerate the model. The client periodically sends cumulative neuron gradients and model weights to the server, and the server prunes the model. To address different requirements, two methods are proposed to determine the pruning rate. The first slowly prunes throughout training, which has limited acceleration during training but can ensure higher accuracy and significantly reduce inference time. The second quickly reaches the target pruning rate early in training and then continues training with a smaller model. This approach may lose more useful information but can complete training faster. Experimental results show that the proposed variational-autoencoder-based federated topic model pruning can greatly accelerate training while maintaining model performance.