mlsanityData to 7 Oct 2026

Investigating semantic subspaces of Transformer sentence embeddings through linear structural probing

arXiv 2310.11923 ↗First posted 18 Oct 2023cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

The question of what kinds of linguistic information are encoded in different layers of Transformer-based language models is of considerable interest for the NLP community. Existing work, however, has overwhelmingly focused on word-level representations and encoder-only language models with the masked-token training objective. In this paper, we present experiments with semantic structural probing, a method for studying sentence-level representations via finding a subspace of the embedding space that provides suitable task-specific pairwise distances between data-points. We apply our method to language models from different families (encoder-only, decoder-only, encoder-decoder) and of different sizes in the context of two tasks, semantic textual similarity and natural-language inference. We find that model families differ substantially in their performance and layer dynamics, but that the results are largely model-size invariant.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 29 of its 47 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

29 of 47 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
17 more
  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  2. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  4. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  5. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  6. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  7. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  8. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  9. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  10. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  11. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 114
  12. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  13. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  14. 2021How is BERT surprised? Layerwise detection of linguistic anomaliescited by 24
  15. 2022An Exploratory Study on Code Attention in BERTcited by 12
  16. 2019Putting words in context: LSTM language models and lexical ambiguitycited by 7
  17. 2022Do LSTMs See Gender? Probing the Ability of LSTMs to Learn Abstract Syntactic Rulescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 3.

12 papers.

  1. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 39
  2. 2025Interpretable Text Embeddings and Text Similarity Explanation: A Surveycited by 12
  3. 2024Are there identifiable structural parts in the sentence embedding whole?cited by 6
  4. 2024Transformer Normalisation Layers and the Independence of Semantic Subspacescited by 6
  5. 2025Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not applycited by 2
  6. 2024A Systematic Survey of Natural Language Processing for the Greek Languagecited by 1
  7. 2025Emergent effects of scaling on the functional hierarchies within large language modelscited by 1
  8. 2025Statistical Deficiency for Task Inclusion Estimationcited by 1
  9. 2026Xetrieval: Mechanistically Explaining Dense Retrievalcited by 1
  10. 2025Enhancing Automatic Term Extraction with Large Language Models via Syntactic Retrievalcited by 0
  11. 2026Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Modelscited by 0
  12. 2026Do LLM Embedding Spaces Recover Expert Structure?cited by 0