# Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings

arXiv: https://arxiv.org/abs/2609.37782 · HTML: https://mlsanity.com/paper/2609.37782/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 29 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [Generative Representational Instruction Tuning](https://mlsanity.com/paper/2402.09906/) (2024)
- [Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents](https://mlsanity.com/paper/2310.19923/) (2023)
- [Scaling Sentence Embeddings with Large Language Models](https://mlsanity.com/paper/2307.16645/) (2023)
- [Repetition Improves Language Model Embeddings](https://mlsanity.com/paper/2402.15449/) (2024)
- [LongEmbed: Extending Embedding Models for Long Context Retrieval](https://mlsanity.com/paper/2404.12096/) (2024)
- [Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics](https://mlsanity.com/paper/2403.01509/) (2024)
- [BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings](https://mlsanity.com/paper/2311.05296/) (2023)
- [Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models](https://mlsanity.com/paper/2404.03921/) (2024)
- [Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT](https://mlsanity.com/paper/2402.07440/) (2024)
- [Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs](https://mlsanity.com/paper/2412.11556/) (2024)
- [Meta-Task Prompting Elicits Embeddings from Large Language Models](https://mlsanity.com/paper/2402.18458/) (2024)
- [GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings](https://mlsanity.com/paper/2410.14635/) (2024)
- [Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering](https://mlsanity.com/paper/2505.12831/) (2025)
- [Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings](https://mlsanity.com/paper/2511.14868/) (2025)
- [LMK > CLS: Landmark Pooling for Dense Embeddings](https://mlsanity.com/paper/2601.21525/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown strong potential as training-free text encoders for long-context embeddings. Existing approaches primarily improve information flow under causal attention and typically construct embeddings by uniformly averaging all token representations. However, for long documents, such mean pooling can dilute salient semantic information with abundant redundant or weakly informative content. To this end, we propose SCSP, a training-free framework that leverages semantic compression for informative token selection in long-context embedding. Specifically, SCSP first partitions a document into sentence-aware chunks and appends a semantic compression prompt to each chunk. A prompt-isolated attention mask preserves information flow among document tokens while restricting each prompt to its corresponding local context. We then use the attention patterns elicited by these prompts to estimate token importance, select informative tokens, and aggregate their intermediate-layer representations into the final embedding. Extensive experiments on long-context embedding benchmarks demonstrate that SCSP can be integrated into both zero-shot and fine-tuned models in a plug-and-play manner, consistently improving their performance.