# Parameter-free Sentence Embedding via Orthogonal Basis

arXiv: https://arxiv.org/abs/1810.00438 · HTML: https://mlsanity.com/paper/1810.00438/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2018.

## Built on by (most cited first, 11 of 11)

- [WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach](https://mlsanity.com/paper/2104.01767/) (2021)
- [HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization](https://mlsanity.com/paper/2106.00149/) (2021)
- [A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations](https://mlsanity.com/paper/2109.04727/) (2021)
- [PAUSE: Positive and Annealed Unlabeled Sentence Embedding](https://mlsanity.com/paper/2109.03155/) (2021)
- [Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation](https://mlsanity.com/paper/2210.09773/) (2022)
- [Discrete Cosine Transform as Universal Sentence Encoder](https://mlsanity.com/paper/2106.00934/) (2021)
- [Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences](https://mlsanity.com/paper/2110.00697/) (2021)
- [Repurposing Knowledge Graph Embeddings for Triple Representation via Weak Supervision](https://mlsanity.com/paper/2208.10328/) (2022)
- [NoPPA: Non-Parametric Pairwise Attention Random Walk Model for Sentence Representation](https://mlsanity.com/paper/2302.12903/) (2023)
- [Static Fuzzy Bag-of-Words: a lightweight sentence embedding algorithm](https://mlsanity.com/paper/2304.03098/) (2023)
- [Sentiment analysis in Tourism: Fine-tuning BERT or sentence embeddings concatenation?](https://mlsanity.com/paper/2312.07797/) (2023)

## Abstract (arXiv)

We propose a simple and robust non-parameterized approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence. We model the semantic meaning of a word in a sentence based on two aspects. One is its relatedness to the word vector subspace already spanned by its contextual words. The other is the word's novel semantic meaning which shall be introduced as a new basis vector perpendicular to this existing subspace. Following this motivation, we develop an innovative method based on orthogonal basis to combine pre-trained word embeddings into sentence representations. This approach requires zero parameters, along with efficient inference performance. We evaluate our approach on 11 downstream NLP tasks. Our model shows superior performance compared with non-parameterized alternatives and it is competitive to other approaches relying on either large amounts of labelled data or prolonged training time.