mlsanityData to 7 Oct 2026

Are there identifiable structural parts in the sentence embedding whole?

arXiv 2406.16563 ↗First posted 24 Jun 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Sentence embeddings from transformer models encode in a fixed length vector much linguistic information. We explore the hypothesis that these embeddings consist of overlapping layers of information that can be separated, and on which specific types of information -- such as information about chunks and their structural and semantic properties -- can be detected. We show that this is the case using a dataset consisting of sentences with known chunk structure, and two linguistic intelligence datasets, solving which relies on detecting chunks and their grammatical number, and respectively, their semantic roles, and through analyses of the performance on the tasks and of the internal representations built during learning.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 67 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

37 of 67 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  10. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  11. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  12. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
25 more
  1. 2019Designing and Interpreting Probes with Control Taskscited by 608
  2. 2015Generating Sentences from a Continuous Spacecited by 607
  3. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  4. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  5. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  6. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  7. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  8. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  9. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  10. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  11. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  12. 2021Low-Complexity Probing via Finding Subnetworkscited by 63
  13. 2019Generating Sentences from Disentangled Syntactic and Semantic Spacescited by 59
  14. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
  15. 2019A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representationscited by 37
  16. 2020What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual modelscited by 35
  17. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  18. 2022SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Featurescited by 33
  19. 2017Deep Semantic Role Labeling with Self-Attentioncited by 23
  20. 2021Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoderscited by 19
  21. 2020Toward Better Storylines with Sentence-Level Language Modelscited by 14
  22. 2022Learning Disentangled Representations for Natural Language Definitionscited by 12
  23. 2023Investigating semantic subspaces of Transformer sentence embeddings through linear structural probingcited by 12
  24. 2023Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Motivations and Formal Specificationscited by 9
  25. 2021Have Attention Heads in BERT Learned Constituency Grammar?cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 2.

6 papers.

  1. 2024Tracking linguistic information in transformer-based sentence embeddings through targeted sparsificationcited by 6
  2. 2024Exploring syntactic information in sentence embeddings through multilingual subject-verb agreementcited by 6
  3. 2024Exploring Italian sentence embeddings properties through multi-taskingcited by 6
  4. 2025Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not applycited by 2
  5. 2026Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Modelscited by 0
  6. 2026Generalization through Lexical Abstraction in Transformer Models: The Case of Functional Wordscited by 0