# SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features

arXiv: https://arxiv.org/abs/2206.07023 · HTML: https://mlsanity.com/paper/2206.07023/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 63 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [A Survey of the State of Explainable AI for Natural Language Processing](https://mlsanity.com/paper/2010.00711/) (2020)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks](https://mlsanity.com/paper/2010.08240/) (2020)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- [Wasserstein Weisfeiler-Lehman Graph Kernels](https://mlsanity.com/paper/1906.01277/) (2019)
- [Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs](https://mlsanity.com/paper/1909.02597/) (2019)
- [Classification and Clustering of Arguments with Contextualized Word Embeddings](https://mlsanity.com/paper/1906.09821/) (2019)
- [SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models](https://mlsanity.com/paper/2002.06652/) (2020)
- [Discretized Integrated Gradients for Explaining Language Models](https://mlsanity.com/paper/2108.13654/) (2021)
- [Gradient-based Analysis of NLP Models is Manipulable](https://mlsanity.com/paper/2010.05419/) (2020)
- [An Incremental Parser for Abstract Meaning Representation](https://mlsanity.com/paper/1608.06111/) (2016)
- [Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR](https://mlsanity.com/paper/2008.08896/) (2020)
- [Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors](https://mlsanity.com/paper/2110.04399/) (2021)
- [Core Semantic First: A Top-down Approach for AMR Parsing](https://mlsanity.com/paper/1909.04303/) (2019)
- [AMR Similarity Metrics from Principles](https://mlsanity.com/paper/2001.10929/) (2020)
- [Picking BERT's Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis](https://mlsanity.com/paper/2011.12073/) (2020)
- [Weisfeiler-Leman in the BAMBOO: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity](https://mlsanity.com/paper/2108.11949/) (2021)
- [Translate, then Parse! A strong baseline for Cross-Lingual AMR Parsing](https://mlsanity.com/paper/2106.04565/) (2021)
- [Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictions](https://mlsanity.com/paper/2109.05853/) (2021)

## Built on by (most cited first, 30 of 33)

- [Improving Contrastive Learning of Sentence Embeddings from AI Feedback](https://mlsanity.com/paper/2305.01918/) (2023)
- [NevIR: Negation in Neural Information Retrieval](https://mlsanity.com/paper/2305.07614/) (2023)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [SMATCH++: Standardized and Extended Evaluation of Semantic Graphs](https://mlsanity.com/paper/2305.06993/) (2023)
- [Natural Language Decompositions of Implicit Content Enable Better Text Representations](https://mlsanity.com/paper/2305.14583/) (2023)
- [Meaning Representations from Trajectories in Autoregressive Models](https://mlsanity.com/paper/2310.18348/) (2023)
- [Towards Explainable Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2306.13041/) (2023)
- [ArcheType: A Novel Framework for Open-Source Column Type Annotation using Large Language Models](https://mlsanity.com/paper/2310.18208/) (2023)
- [Interpretable Text Embeddings and Text Similarity Explanation: A Survey](https://mlsanity.com/paper/2502.14862/) (2025)
- [Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning](https://mlsanity.com/paper/2506.08354/) (2025)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [An Attribution Method for Siamese Encoders](https://mlsanity.com/paper/2310.05703/) (2023)
- [Representation biases in sentence transformers](https://mlsanity.com/paper/2301.13039/) (2023)
- [Description-Based Text Similarity](https://mlsanity.com/paper/2305.12517/) (2023)
- [Approximate Attributions for Off-the-Shelf Siamese Transformers](https://mlsanity.com/paper/2402.02883/) (2024)
- [Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?](https://mlsanity.com/paper/2502.04718/) (2025)
- [Are there identifiable structural parts in the sentence embedding whole?](https://mlsanity.com/paper/2406.16563/) (2024)
- [Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification](https://mlsanity.com/paper/2407.18119/) (2024)
- [PRISM: A Framework for Producing Interpretable Political Bias Embeddings with Political-Aware Cross-Encoder](https://mlsanity.com/paper/2505.24646/) (2025)
- [Sentence Smith: Controllable Edits for Evaluating Text Embeddings](https://mlsanity.com/paper/2502.14734/) (2025)
- [Onco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncology](https://mlsanity.com/paper/2404.06680/) (2024)
- [WordDecipher: Enhancing Digital Workspace Communication with Explainable AI for Non-native English Speakers](https://mlsanity.com/paper/2404.07005/) (2024)
- [Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply](https://mlsanity.com/paper/2509.01606/) (2025)
- [SBASH: a Framework for Designing and Evaluating RAG vs. Prompt-Tuned LLM Honeypots](https://mlsanity.com/paper/2510.21459/) (2025)
- [Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol](https://mlsanity.com/paper/2602.13320/) (2026)
- [Space Decomposition for Sentence Embedding](https://mlsanity.com/paper/2406.03125/) (2024)
- [ZeroDL: Zero-shot Distribution Learning for Text Clustering via Large Language Models](https://mlsanity.com/paper/2406.13342/) (2024)
- [Xetrieval: Mechanistically Explaining Dense Retrieval](https://mlsanity.com/paper/2605.29507/) (2026)
- [Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures](https://mlsanity.com/paper/2305.18915/) (2023)
- [AMR4NLI: Interpretable and robust NLI measures from semantic graphs](https://mlsanity.com/paper/2306.00936/) (2023)

## Abstract (arXiv)

Models based on large-pretrained language models, such as S(entence)BERT, provide effective and efficient sentence embeddings that show high correlation to human similarity ratings, but lack interpretability. On the other hand, graph metrics for graph-based meaning representations (e.g., Abstract Meaning Representation, AMR) can make explicit the semantic aspects in which two sentences are similar. However, such metrics tend to be slow, rely on parsers, and do not reach state-of-the-art performance when rating sentence similarity. In this work, we aim at the best of both worlds, by learning to induce Semantically Structured Sentence BERT embeddings (S3BERT). Our S3BERT embeddings are composed of explainable sub-embeddings that emphasize various semantic sentence features (e.g., semantic roles, negation, or quantification). We show how to i) learn a decomposition of the sentence embeddings into semantic features, through approximation of a suite of interpretable AMR graph metrics, and how to ii) preserve the overall power of the neural embeddings by controlling the decomposition learning process with a second objective that enforces consistency with the similarity ratings of an SBERT teacher model. In our experimental studies, we show that our approach offers interpretability -- while fully preserving the effectiveness and efficiency of the neural sentence embeddings.