# Unsupervised Sentence Representations as Word Information Series: Revisiting TF--IDF

arXiv: https://arxiv.org/abs/1710.06524 · HTML: https://mlsanity.com/paper/1710.06524/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2017.

## Built on by (most cited first, 6 of 6)

- [A Review of the Trends and Challenges in Adopting Natural Language Processing Methods for Education Feedback Analysis](https://mlsanity.com/paper/2301.08826/) (2023)
- [Generative User-Experience Research for Developing Domain-specific Natural Language Processing Applications](https://mlsanity.com/paper/2306.16143/) (2023)
- [Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words](https://mlsanity.com/paper/2310.15921/) (2023)
- [Back to the Basics: A Quantitative Analysis of Statistical and Graph-Based Term Weighting Schemes for Keyword Extraction](https://mlsanity.com/paper/2104.08028/) (2021)
- [LUC at ComMA-2021 Shared Task: Multilingual Gender Biased and Communal Language Identification without using linguistic features](https://mlsanity.com/paper/2112.10189/) (2021)
- [Common Sense Knowledge Learning for Open Vocabulary Neural Reasoning: A First View into Chronic Disease Literature](https://mlsanity.com/paper/2111.13781/) (2021)

## Abstract (arXiv)

Sentence representation at the semantic level is a challenging task for Natural Language Processing and Artificial Intelligence. Despite the advances in word embeddings (i.e. word vector representations), capturing sentence meaning is an open question due to complexities of semantic interactions among words. In this paper, we present an embedding method, which is aimed at learning unsupervised sentence representations from unlabeled text. We propose an unsupervised method that models a sentence as a weighted series of word embeddings. The weights of the word embeddings are fitted by using Shannon's word entropies provided by the Term Frequency--Inverse Document Frequency (TF--IDF) transform. The hyperparameters of the model can be selected according to the properties of data (e.g. sentence length and textual gender). Hyperparameter selection involves word embedding methods and dimensionalities, as well as weighting schemata. Our method offers advantages over existing methods: identifiable modules, short-term training, online inference of (unseen) sentence representations, as well as independence from domain, external knowledge and language resources. Results showed that our model outperformed the state of the art in well-known Semantic Textual Similarity (STS) benchmarks. Moreover, our model reached state-of-the-art performance when compared to supervised and knowledge-based STS systems.