# Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks

arXiv: https://arxiv.org/abs/1608.04207 · HTML: https://mlsanity.com/paper/1608.04207/

## Key facts

- Cited by 216 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2016.

## Built on by (most cited first, 30 of 216)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://mlsanity.com/paper/2402.05162/) (2024)
- [Post-hoc Interpretability for Neural NLP: A Survey](https://mlsanity.com/paper/2108.04840/) (2021)
- [Analyzing Transformers in Embedding Space](https://mlsanity.com/paper/2209.02535/) (2022)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction](https://mlsanity.com/paper/2312.13558/) (2023)
- [On Memorization of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2410.23123/) (2024)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [Jump to Conclusions: Short-Cutting Transformers With Linear Transformations](https://mlsanity.com/paper/2303.09435/) (2023)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [Chess as a Testbed for Language Model State Tracking](https://mlsanity.com/paper/2102.13249/) (2021)
- [First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT](https://mlsanity.com/paper/2101.11109/) (2021)
- [Low-Complexity Probing via Finding Subnetworks](https://mlsanity.com/paper/2104.03514/) (2021)
- [Conditional probing: measuring usable information beyond a baseline](https://mlsanity.com/paper/2109.09234/) (2021)
- [On the Pitfalls of Analyzing Individual Neurons in Language Models](https://mlsanity.com/paper/2110.07483/) (2021)
- [Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP](https://mlsanity.com/paper/2308.14179/) (2023)
- [Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction](https://mlsanity.com/paper/2105.06965/) (2021)
- [Probing for the Usage of Grammatical Number](https://mlsanity.com/paper/2204.08831/) (2022)
- [Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?](https://mlsanity.com/paper/2104.10809/) (2021)
- [The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models](https://mlsanity.com/paper/2310.11877/) (2023)
- [CausalGym: Benchmarking causal interpretability methods on linguistic tasks](https://mlsanity.com/paper/2402.12560/) (2024)
- [Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration](https://mlsanity.com/paper/2109.06304/) (2021)
- [SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis](https://mlsanity.com/paper/2302.00785/) (2023)
- [Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks](https://mlsanity.com/paper/2201.09745/) (2022)
- [Probing Classifiers are Unreliable for Concept Removal and Detection](https://mlsanity.com/paper/2207.04153/) (2022)
- [Estimating Knowledge in Large Language Models Without Generating a Single Token](https://mlsanity.com/paper/2406.12673/) (2024)

## Abstract (arXiv)

There is a lot of research interest in encoding variable length sentences into fixed length vectors, in a way that preserves the sentence meanings. Two common methods include representations based on averaging word vectors, and representations based on the hidden states of recurrent neural networks such as LSTMs. The sentence vectors are used as features for subsequent machine learning tasks or for pre-training in the context of deep learning. However, not much is known about the properties that are encoded in these sentence representations and about the language information they capture. We propose a framework that facilitates better understanding of the encoded representations. We define prediction tasks around isolated aspects of sentence structure (namely sentence length, word content, and word order), and score representations by the ability to train a classifier to solve each prediction task when using the representation as input. We demonstrate the potential contribution of the approach by analyzing different sentence representation mechanisms. The analysis sheds light on the relative strengths of different sentence embedding methods with respect to these low level prediction tasks, and on the effect of the encoded vector's dimensionality on the resulting representations.