# Analyzing Text Representations by Measuring Task Alignment

arXiv: https://arxiv.org/abs/2305.19747 · HTML: https://mlsanity.com/paper/2305.19747/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 45 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [Analyzing and Improving Representations with the Soft Nearest Neighbor Loss](https://mlsanity.com/paper/1902.01889/) (2019)
- [A Survey of Active Learning for Text Classification using Deep Neural Networks](https://mlsanity.com/paper/2008.07267/) (2020)
- [DirectProbe: Studying Representations without Classifiers](https://mlsanity.com/paper/2104.05904/) (2021)
- [Probing Multilingual Sentence Representations With X-Probe](https://mlsanity.com/paper/1906.05061/) (2019)
- [Cluster & Tune: Boost Cold Start Performance in Text Classification](https://mlsanity.com/paper/2203.10581/) (2022)
- [Comparing Text Representations: A Theory-Driven Approach](https://mlsanity.com/paper/2109.07458/) (2021)
- [The Benefits of Word Embeddings Features for Active Learning in Clinical Information Extraction](https://mlsanity.com/paper/1607.02810/) (2016)
- [Investigating the Effectiveness of Representations Based on Word-Embeddings in Active Learning for Labelling Text Datasets](https://mlsanity.com/paper/1910.03505/) (2019)

## Built on by (most cited first, 2 of 2)

- [Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings](https://mlsanity.com/paper/2510.19694/) (2025)
- [Domain Pre-training Impact on Representations](https://mlsanity.com/paper/2505.24455/) (2025)

## Abstract (arXiv)

Textual representations based on pre-trained language models are key, especially in few-shot learning scenarios. What makes a representation good for text classification? Is it due to the geometric properties of the space or because it is well aligned with the task? We hypothesize the second claim. To test it, we develop a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity. Our experiments on text classification validate our hypothesis by showing that task alignment can explain the classification performance of a given representation.