# Span-Aggregatable, Contextualized Word Embeddings for Effective Phrase Mining

arXiv: https://arxiv.org/abs/2405.07263 · HTML: https://mlsanity.com/paper/2405.07263/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 12 May 2024.

## Builds on (most cited first, 11 of 11)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval](https://mlsanity.com/paper/2109.10086/) (2021)
- [Learning Dense Representations of Phrases at Scale](https://mlsanity.com/paper/2012.12624/) (2020)
- [Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration](https://mlsanity.com/paper/2109.06304/) (2021)
- [Assessing Phrasal Representation and Composition in Transformers](https://mlsanity.com/paper/2010.03763/) (2020)
- [PiC: A Phrase-in-Context Dataset for Phrase Understanding and Semantic Search](https://mlsanity.com/paper/2207.09068/) (2022)

## Built on by (most cited first, 1 of 1)

- [Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches](https://mlsanity.com/paper/2503.11376/) (2025)

## Abstract (arXiv)

Dense vector representations for sentences made significant progress in recent years as can be seen on sentence similarity tasks. Real-world phrase retrieval applications, on the other hand, still encounter challenges for effective use of dense representations. We show that when target phrases reside inside noisy context, representing the full sentence with a single dense vector, is not sufficient for effective phrase retrieval. We therefore look into the notion of representing multiple, sub-sentence, consecutive word spans, each with its own dense vector. We show that this technique is much more effective for phrase mining, yet requires considerable compute to obtain useful span representations. Accordingly, we make an argument for contextualized word/token embeddings that can be aggregated for arbitrary word spans while maintaining the span's semantic meaning. We introduce a modification to the common contrastive loss used for sentence embeddings that encourages word embeddings to have this property. To demonstrate the effect of this method we present a dataset based on the STS-B dataset with additional generated text, that requires finding the best matching paraphrase residing in a larger context and report the degree of similarity to the origin phrase. We demonstrate on this dataset, how our proposed method can achieve better results without significant increase to compute.