# Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms

arXiv: https://arxiv.org/abs/1805.09843 · HTML: https://mlsanity.com/paper/1805.09843/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2018.

## Built on by (most cited first, 30 of 53)

- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Parallelizing Legendre Memory Unit Training](https://mlsanity.com/paper/2102.11417/) (2021)
- [Sparse Autoencoder Features for Classifications and Transferability](https://mlsanity.com/paper/2502.11367/) (2025)
- [Subword Pooling Makes a Difference](https://mlsanity.com/paper/2102.10864/) (2021)
- [Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP](https://mlsanity.com/paper/2109.03777/) (2021)
- [Sparse Structure Learning via Graph Neural Networks for Inductive Document Classification](https://mlsanity.com/paper/2112.06386/) (2021)
- [Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention](https://mlsanity.com/paper/2106.04133/) (2021)
- [Billion-scale Pre-trained E-commerce Product Knowledge Graph Model](https://mlsanity.com/paper/2105.00388/) (2021)
- [Paraphrase Identification with Deep Learning: A Review of Datasets and Methods](https://mlsanity.com/paper/2212.06933/) (2022)
- [DeepCVA: Automated Commit-level Vulnerability Assessment with Deep Multi-task Learning](https://mlsanity.com/paper/2108.08041/) (2021)
- [Training Effective Neural Sentence Encoders from Automatically Mined Paraphrases](https://mlsanity.com/paper/2207.12759/) (2022)
- [Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings](https://mlsanity.com/paper/2506.01435/) (2025)
- [Enhancing Interpretable Clauses Semantically using Pretrained Word Representation](https://mlsanity.com/paper/2104.06901/) (2021)
- [Re-evaluating Word Mover's Distance](https://mlsanity.com/paper/2105.14403/) (2021)
- [CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling](https://mlsanity.com/paper/2210.07661/) (2022)
- [Clustering Word Embeddings with Self-Organizing Maps. Application on LaRoSeDa -- A Large Romanian Sentiment Data Set](https://mlsanity.com/paper/2101.04197/) (2021)
- [FreSaDa: A French Satire Data Set for Cross-Domain Satire Detection](https://mlsanity.com/paper/2104.04828/) (2021)
- [Less is More: Parameter-Free Text Classification with Gzip](https://mlsanity.com/paper/2212.09410/) (2022)
- [Retrieving Black-box Optimal Images from External Databases](https://mlsanity.com/paper/2112.14921/) (2021)
- [Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2305.18824/) (2023)
- [Using AntiPatterns to avoid MLOps Mistakes](https://mlsanity.com/paper/2107.00079/) (2021)
- [On the Use of Fine-grained Vulnerable Code Statements for Software Vulnerability Assessment Models](https://mlsanity.com/paper/2203.08417/) (2022)
- [Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition](https://mlsanity.com/paper/2606.00959/) (2026)
- [kogito: A Commonsense Knowledge Inference Toolkit](https://mlsanity.com/paper/2211.08451/) (2022)
- [Deep Learning-based Sentiment Classification: A Comparative Survey](https://mlsanity.com/paper/2312.17253/) (2023)
- [Document Embedding for Scientific Articles: Efficacy of Word Embeddings vs TFIDF](https://mlsanity.com/paper/2107.05151/) (2021)
- [The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation](https://mlsanity.com/paper/2205.06618/) (2022)
- [Expanding Knowledge Graphs with Humans in the Loop](https://mlsanity.com/paper/2212.05189/) (2022)
- [Hierarchical information matters: Text classification via tree based graph neural network](https://mlsanity.com/paper/2110.02047/) (2021)
- [Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings](https://mlsanity.com/paper/2604.27398/) (2026)

## Abstract (arXiv)

Many deep learning architectures have been proposed to model the compositionality in text sequences, requiring a substantial number of parameters and expensive computations. However, there has not been a rigorous evaluation regarding the added value of sophisticated compositional functions. In this paper, we conduct a point-by-point comparative study between Simple Word-Embedding-based Models (SWEMs), consisting of parameter-free pooling operations, relative to word-embedding-based RNN/CNN models. Surprisingly, SWEMs exhibit comparable or even superior performance in the majority of cases considered. Based upon this understanding, we propose two additional pooling strategies over learned word embeddings: (i) a max-pooling operation for improved interpretability; and (ii) a hierarchical pooling operation, which preserves spatial (n-gram) information within text sequences. We present experiments on 17 datasets encompassing three tasks: (i) (long) document classification; (ii) text sequence matching; and (iii) short text tasks, including classification and tagging. The source code and datasets can be obtained from https:// github.com/dinghanshen/SWEM.