# Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors

arXiv: https://arxiv.org/abs/1904.13264 · HTML: https://mlsanity.com/paper/1904.13264/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2019.

## Built on by (most cited first, 7 of 7)

- [Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings](https://mlsanity.com/paper/2106.14361/) (2021)
- [Representation biases in sentence transformers](https://mlsanity.com/paper/2301.13039/) (2023)
- [Unsupervised Sentence Textual Similarity with Compositional Phrase Semantics](https://mlsanity.com/paper/2210.02284/) (2022)
- [Improving word mover's distance by leveraging self-attention matrix](https://mlsanity.com/paper/2211.06229/) (2022)
- [Subspace Representations for Soft Set Operations and Sentence Similarities](https://mlsanity.com/paper/2210.13034/) (2022)
- [SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2404.17606/) (2024)
- [Static Fuzzy Bag-of-Words: a lightweight sentence embedding algorithm](https://mlsanity.com/paper/2304.03098/) (2023)

## Abstract (arXiv)

Recent literature suggests that averaged word vectors followed by simple post-processing outperform many deep learning methods on semantic textual similarity tasks. Furthermore, when averaged word vectors are trained supervised on large corpora of paraphrases, they achieve state-of-the-art results on standard STS benchmarks. Inspired by these insights, we push the limits of word embeddings even further. We propose a novel fuzzy bag-of-words (FBoW) representation for text that contains all the words in the vocabulary simultaneously but with different degrees of membership, which are derived from similarities between word vectors. We show that max-pooled word vectors are only a special case of fuzzy BoW and should be compared via fuzzy Jaccard index rather than cosine similarity. Finally, we propose DynaMax, a completely unsupervised and non-parametric similarity measure that dynamically extracts and max-pools good features depending on the sentence pair. This method is both efficient and easy to implement, yet outperforms current baselines on STS tasks by a large margin and is even competitive with supervised word vectors trained to directly optimise cosine similarity.