mlsanityData to 7 Oct 2026

NoPPA: Non-Parametric Pairwise Attention Random Walk Model for Sentence Representation

arXiv 2302.12903 ↗First posted 24 Feb 2023cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose a novel non-parametric/un-trainable language model, named Non-Parametric Pairwise Attention Random Walk Model (NoPPA), to generate sentence embedding only with pre-trained word embedding and pre-counted word frequency. To the best we know, this study is the first successful attempt to break the constraint on bag-of-words assumption with a non-parametric attention mechanism. We evaluate our method on eight different downstream classification tasks. The experiment results show that NoPPA outperforms all kinds of bag-of-words-based methods in each dataset and provides a comparable or better performance than the state-of-the-art non-parametric methods on average. Furthermore, visualization supports that NoPPA can understand contextual topics, common phrases, and word causalities. Our model is available at https://github.com/JacksonWuxs/NoPPA.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 24 Feb 2023.

Builds on

29 of 46 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  9. 2016FastText.zip: Compressing text classification modelscited by 331
  10. 2015Skip-Thought Vectorscited by 321
  11. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  12. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
17 more
  1. 2018An efficient framework for learning sentence representationscited by 197
  2. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
  3. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  4. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  5. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  6. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  7. 2015Towards Universal Paraphrastic Sentence Embeddingscited by 59
  8. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 55
  9. 2018Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representationscited by 20
  10. 2021Smoothed Contrastive Learning for Unsupervised Sentence Embeddingcited by 14
  11. 2019Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embeddingcited by 12
  12. 2018Parameter-free Sentence Embedding via Orthogonal Basiscited by 11
  13. 2019Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representationcited by 10
  14. 2019Efficient Sentence Embedding using Discrete Cosine Transformcited by 7
  15. 2021Dual-View Distilled BERT for Sentence Embeddingcited by 7
  16. 2016Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representationscited by 5
  17. 2020Efficient Sentence Embedding via Semantic Subspace Analysiscited by 4

Built on by

Not cited yet.