# NoPPA: Non-Parametric Pairwise Attention Random Walk Model for Sentence Representation

arXiv: https://arxiv.org/abs/2302.12903 · HTML: https://mlsanity.com/paper/2302.12903/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 24 Feb 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [Learning Distributed Representations of Sentences from Unlabelled Data](https://mlsanity.com/paper/1602.03483/) (2016)
- [On the Sentence Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2011.05864/) (2020)
- [PromptBERT: Improving BERT Sentence Embeddings with Prompts](https://mlsanity.com/paper/2201.04337/) (2022)
- [WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach](https://mlsanity.com/paper/2104.01767/) (2021)
- [Towards Universal Paraphrastic Sentence Embeddings](https://mlsanity.com/paper/1511.08198/) (2015)
- [SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models](https://mlsanity.com/paper/2002.06652/) (2020)
- [Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations](https://mlsanity.com/paper/1803.01400/) (2018)
- [Smoothed Contrastive Learning for Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04321/) (2021)
- [Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding](https://mlsanity.com/paper/1908.05161/) (2019)
- [Parameter-free Sentence Embedding via Orthogonal Basis](https://mlsanity.com/paper/1810.00438/) (2018)
- [Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation](https://mlsanity.com/paper/1902.08850/) (2019)
- [Efficient Sentence Embedding using Discrete Cosine Transform](https://mlsanity.com/paper/1909.03104/) (2019)
- [Dual-View Distilled BERT for Sentence Embedding](https://mlsanity.com/paper/2104.08675/) (2021)
- [Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations](https://mlsanity.com/paper/1606.00819/) (2016)
- [Efficient Sentence Embedding via Semantic Subspace Analysis](https://mlsanity.com/paper/2002.09620/) (2020)

## Abstract (arXiv)

We propose a novel non-parametric/un-trainable language model, named Non-Parametric Pairwise Attention Random Walk Model (NoPPA), to generate sentence embedding only with pre-trained word embedding and pre-counted word frequency. To the best we know, this study is the first successful attempt to break the constraint on bag-of-words assumption with a non-parametric attention mechanism. We evaluate our method on eight different downstream classification tasks. The experiment results show that NoPPA outperforms all kinds of bag-of-words-based methods in each dataset and provides a comparable or better performance than the state-of-the-art non-parametric methods on average. Furthermore, visualization supports that NoPPA can understand contextual topics, common phrases, and word causalities. Our model is available at https://github.com/JacksonWuxs/NoPPA.