# Investigating Distributions of Telecom Adapted Sentence Embeddings for Document Retrieval

arXiv: https://arxiv.org/abs/2406.12336 · HTML: https://mlsanity.com/paper/2406.12336/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [A Multi-Task Embedder For Retrieval Augmented LLMs](https://mlsanity.com/paper/2310.07554/) (2023)
- [IsoScore: Measuring the Uniformity of Embedding Space Utilization](https://mlsanity.com/paper/2108.07344/) (2021)
- [Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words](https://mlsanity.com/paper/2205.05092/) (2022)
- [On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers](https://mlsanity.com/paper/2010.02616/) (2020)
- [The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models](https://mlsanity.com/paper/2311.05928/) (2023)
- [How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy](https://mlsanity.com/paper/2109.04740/) (2021)
- [Stable Anisotropic Regularization](https://mlsanity.com/paper/2305.19358/) (2023)
- [Isotropic Representation Can Improve Dense Retrieval](https://mlsanity.com/paper/2209.00218/) (2022)
- [Is Anisotropy Inherent to Transformers?](https://mlsanity.com/paper/2306.07656/) (2023)
- [A Comparative Study of Sentence Embedding Models for Assessing Semantic Variation](https://mlsanity.com/paper/2308.04625/) (2023)

## Built on by (most cited first, 1 of 1)

- [T-VEC: A Telecom-Specific Vectorization Model with Enhanced Semantic Understanding via Deep Triplet Loss Fine-Tuning](https://mlsanity.com/paper/2504.16460/) (2025)

## Abstract (arXiv)

A plethora of sentence embedding models makes it challenging to choose one, especially for technical domains rich with specialized vocabulary. In this work, we domain adapt embeddings using telecom data for question answering. We evaluate embeddings obtained from publicly available models and their domain-adapted variants, on both point retrieval accuracies, as well as their (95%) confidence intervals. We establish a systematic method to obtain thresholds for similarity scores for different embeddings. As expected, we observe that fine-tuning improves mean bootstrapped accuracies. We also observe that it results in tighter confidence intervals, which further improve when pre-training is preceded by fine-tuning. We introduce metrics which measure the distributional overlaps of top-K, correct and random document similarities with the question. Further, we show that these metrics are correlated with retrieval accuracy and similarity thresholds. Recent literature shows conflicting effects of isotropy on retrieval accuracies. Our experiments establish that the isotropy of embeddings (as measured by two independent state-of-the-art isotropy metric definitions) is poorly correlated with retrieval performance. We show that embeddings for domain-specific sentences have little overlap with those for domain-agnostic ones, and fine-tuning moves them further apart. Based on our results, we provide recommendations for use of our methodology and metrics by researchers and practitioners.