# Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables

arXiv: https://arxiv.org/abs/2211.03616 · HTML: https://mlsanity.com/paper/2211.03616/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable](https://mlsanity.com/paper/1910.07931/) (2019)
- [Bilateral Multi-Perspective Matching for Natural Language Sentences](https://mlsanity.com/paper/1702.03814/) (2017)
- [Semantics-aware BERT for Language Understanding](https://mlsanity.com/paper/1909.02209/) (2019)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Convolutional Neural Network Architectures for Matching Natural Language Sentences](https://mlsanity.com/paper/1503.03244/) (2015)
- [Discovering Discrete Latent Topics with Neural Variational Inference](https://mlsanity.com/paper/1706.00359/) (2017)
- [Question Answering with Subgraph Embeddings](https://mlsanity.com/paper/1406.3676/) (2014)
- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [Neural Topic Model via Optimal Transport](https://mlsanity.com/paper/2008.13537/) (2020)
- [Neural Natural Language Inference Models Enhanced with External Knowledge](https://mlsanity.com/paper/1711.04289/) (2017)
- [ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs](https://mlsanity.com/paper/1512.05193/) (2015)
- [Topic-Aware Neural Keyphrase Generation for Social Media Language](https://mlsanity.com/paper/1906.03889/) (2019)
- [Topic Memory Networks for Short Text Classification](https://mlsanity.com/paper/1809.03664/) (2018)
- [Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks](https://mlsanity.com/paper/2102.10934/) (2021)
- [SemEval-2017 Task 3: Community Question Answering](https://mlsanity.com/paper/1912.00730/) (2019)
- [SemEval-2016 Task 3: Community Question Answering](https://mlsanity.com/paper/1912.01972/) (2019)
- [SemEval-2015 Task 3: Answer Selection in Community Question Answering](https://mlsanity.com/paper/1911.11403/) (2019)
- [Discrete Latent Variable Representations for Low-Resource Text Classification](https://mlsanity.com/paper/2006.06226/) (2020)
- [Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncoder](https://mlsanity.com/paper/2006.08101/) (2020)

## Abstract (arXiv)

Recently, discrete latent variable models have received a surge of interest in both Natural Language Processing (NLP) and Computer Vision (CV), attributed to their comparable performance to the continuous counterparts in representation learning, while being more interpretable in their predictions. In this paper, we develop a topic-informed discrete latent variable model for semantic textual similarity, which learns a shared latent space for sentence-pair representation via vector quantization. Compared with previous models limited to local semantic contexts, our model can explore richer semantic information via topic modeling. We further boost the performance of semantic similarity by injecting the quantized representation into a transformer-based language model with a well-designed semantic-driven attention mechanism. We demonstrate, through extensive experiments across various English language datasets, that our model is able to surpass several strong neural baselines in semantic textual similarity tasks.