# Principles of Concept Representation in Sentence Encoders

arXiv: https://arxiv.org/abs/2606.06994 · HTML: https://mlsanity.com/paper/2606.06994/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Builds on (most cited first, 13 of 13)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Self-Alignment Pretraining for Biomedical Entity Representations](https://mlsanity.com/paper/2010.11784/) (2020)
- [Biomedical Entity Representations with Synonym Marginalization](https://mlsanity.com/paper/2005.00239/) (2020)
- [Learning to Understand Phrases by Embedding the Dictionary](https://mlsanity.com/paper/1504.00548/) (2015)
- [Assessing Composition in Sentence Vector Representations](https://mlsanity.com/paper/1809.03992/) (2018)
- [Learning Disentangled Representations for Natural Language Definitions](https://mlsanity.com/paper/2210.02898/) (2022)
- [Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitions](https://mlsanity.com/paper/2305.07303/) (2023)
- [A Systematic Comparison of English Noun Compound Representations](https://mlsanity.com/paper/1906.04772/) (2019)

## Abstract (arXiv)

What makes a sentence encoder produce good concept representations? We approach this through the lens of representational compositionality: an encoder supports a concept family only when its latent space admits a low-distortion realization of the corresponding semantic operator. This framing predicts both where current encoders succeed and where they are structurally mismatched to their supervision. Through a controlled ablation over encoder conditions trained on 3.3 million synonym and definition pairs from WordNet and Wiktionary, evaluated on three decontaminated splits and a modifier-labeled noun-phrase benchmark, we identify four principles. Fine-tuning recalibrates the latent geometry rather than expanding it (P1). Semantic signal concentrates in the final transformer layer before concept-specific training begins, making cross-layer pooling redundant (P2). Hard negatives improve discrimination and stress-test robustness without improving retrieval ranking, showing that calibration and ranking are independently addressable (P3). Finally, the effectiveness of supervision depends on the composition type of the target concept. Extensional training helps intersective and subsective families while degrading relational and intensional ones, exposing a structural limitation of current training paradigms (P4). We release two new evaluation datasets: a DBpedia semantic-gap benchmark and a modifier-labeled NP paraphrase suite.