# Universal Text Representation from BERT: An Empirical Study

arXiv: https://arxiv.org/abs/1910.07973 · HTML: https://mlsanity.com/paper/1910.07973/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2019.

## Built on by (most cited first, 28 of 28)

- [BERT: A Review of Applications in Natural Language Processing and Understanding](https://mlsanity.com/paper/2103.11943/) (2021)
- [GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning](https://mlsanity.com/paper/2402.16829/) (2024)
- [A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change](https://mlsanity.com/paper/2402.12011/) (2024)
- [Prompt-Learning for Short Text Classification](https://mlsanity.com/paper/2202.11345/) (2022)
- [L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathi](https://mlsanity.com/paper/2211.11187/) (2022)
- [Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning](https://mlsanity.com/paper/2104.01666/) (2021)
- [Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining](https://mlsanity.com/paper/2105.10419/) (2021)
- [Few-Shot Stance Detection via Target-Aware Prompt Distillation](https://mlsanity.com/paper/2206.13214/) (2022)
- [Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning](https://mlsanity.com/paper/2406.11890/) (2024)
- [(Chat)GPT v BERT: Dawn of Justice for Semantic Change Detection](https://mlsanity.com/paper/2401.14040/) (2024)
- [Ontology-Driven and Weakly Supervised Rare Disease Identification from Clinical Notes](https://mlsanity.com/paper/2205.05656/) (2022)
- [Representation biases in sentence transformers](https://mlsanity.com/paper/2301.13039/) (2023)
- [Learning by Hallucinating: Vision-Language Pre-training with Weak Supervision](https://mlsanity.com/paper/2210.13591/) (2022)
- [Prompt-tuning for Clickbait Detection via Text Summarization](https://mlsanity.com/paper/2404.11206/) (2024)
- [Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models](https://mlsanity.com/paper/2601.20546/) (2026)
- [Don't be a Fool: Pooling Strategies in Offensive Language Detection from User-Intended Adversarial Attacks](https://mlsanity.com/paper/2403.15467/) (2024)
- [Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models](https://mlsanity.com/paper/2410.12011/) (2024)
- [BERTVision -- A Parameter-Efficient Approach for Question Answering](https://mlsanity.com/paper/2202.12210/) (2022)
- [Sentence Ambiguity, Grammaticality and Complexity Probes](https://mlsanity.com/paper/2210.06928/) (2022)
- [CoSD: Collaborative Stance Detection with Contrastive Heterogeneous Topic Graph Learning](https://mlsanity.com/paper/2404.17609/) (2024)
- [Analyzing Narrative Processing in Large Language Models (LLMs): Using GPT4 to test BERT](https://mlsanity.com/paper/2405.02024/) (2024)
- [CMLCompiler: A Unified Compiler for Classical Machine Learning](https://mlsanity.com/paper/2301.13441/) (2023)
- [Domain Adaptation of Multilingual Semantic Search -- Literature Review](https://mlsanity.com/paper/2402.02932/) (2024)
- [VI-OOD: A Unified Representation Learning Framework for Textual Out-of-distribution Detection](https://mlsanity.com/paper/2404.06217/) (2024)
- [Enhancing BERTopic with Intermediate Layer Representations](https://mlsanity.com/paper/2505.06696/) (2025)
- [HyperPersona: A Multi-Level Hypergraph Framework for Text-Based Automatic Personality Prediction](https://mlsanity.com/paper/2605.17355/) (2026)
- [Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perception](https://mlsanity.com/paper/2607.08321/) (2026)
- [Gradient-free Task-Conditioned Retrieval for On-Device In-Context Learning](https://mlsanity.com/paper/2607.27766/) (2026)

## Abstract (arXiv)

We present a systematic investigation of layer-wise BERT activations for general-purpose text representations to understand what linguistic information they capture and how transferable they are across different tasks. Sentence-level embeddings are evaluated against two state-of-the-art models on downstream and probing tasks from SentEval, while passage-level embeddings are evaluated on four question-answering (QA) datasets under a learning-to-rank problem setting. Embeddings from the pre-trained BERT model perform poorly in semantic similarity and sentence surface information probing tasks. Fine-tuning BERT on natural language inference data greatly improves the quality of the embeddings. Combining embeddings from different BERT layers can further boost performance. BERT embeddings outperform BM25 baseline significantly on factoid QA datasets at the passage level, but fail to perform better than BM25 on non-factoid datasets. For all QA datasets, there is a gap between embedding-based method and in-domain fine-tuned BERT (we report new state-of-the-art results on two datasets), which suggests deep interactions between question and answer pairs are critical for those hard tasks.