# Self-Supervised Learning for Neural Topic Models with Variance-Invariance-Covariance Regularization

arXiv: https://arxiv.org/abs/2502.09944 · HTML: https://mlsanity.com/paper/2502.09944/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Discovering Discrete Latent Topics with Neural Variational Inference](https://mlsanity.com/paper/1706.00359/) (2017)
- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [Contrastive Learning for Neural Topic Model](https://mlsanity.com/paper/2110.12764/) (2021)
- [Neural Models for Documents with Metadata](https://mlsanity.com/paper/1705.09296/) (2017)
- [Effective Neural Topic Modeling with Embedding Clustering Regularization](https://mlsanity.com/paper/2306.04217/) (2023)
- [Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning](https://mlsanity.com/paper/2211.12878/) (2022)
- [TeachAugment: Data Augmentation Optimization Using Teacher Knowledge](https://mlsanity.com/paper/2202.12513/) (2022)
- [Improving Neural Topic Models using Knowledge Distillation](https://mlsanity.com/paper/2010.02377/) (2020)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)

## Abstract (arXiv)

In our study, we propose a self-supervised neural topic model (NTM) that combines the power of NTMs and regularized self-supervised learning methods to improve performance. NTMs use neural networks to learn latent topics hidden behind the words in documents, enabling greater flexibility and the ability to estimate more coherent topics compared to traditional topic models. On the other hand, some self-supervised learning methods use a joint embedding architecture with two identical networks that produce similar representations for two augmented versions of the same input. Regularizations are applied to these representations to prevent collapse, which would otherwise result in the networks outputting constant or redundant representations for all inputs. Our model enhances topic quality by explicitly regularizing latent topic representations of anchor and positive samples. We also introduced an adversarial data augmentation method to replace the heuristic sampling method. We further developed several variation models including those on the basis of an NTM that incorporates contrastive learning with both positive and negative samples. Experimental results on three datasets showed that our models outperformed baselines and state-of-the-art models both quantitatively and qualitatively.