Training for Compositional Sensitivity Reduces Dense Retrieval Generalization
Not on the map. Why?
Abstract (arXiv)
Dense retrieval compresses texts into single embeddings ranked by cosine similarity. While efficient for recall, this interface is brittle for identity-level matching: minimal compositional edits (negation, role swaps) flip meaning yet retain high similarity. Motivated by geometric results for unit-sphere cosine spaces (Kang et al., 2025), we test this retrieval-composition tension in text-only retrieval. Across four dual-encoder backbones, adding structure-targeted negatives consistently reduces zero-shot NanoBEIR retrieval (8-9% mean nDCG@10 drop on small backbones; up to 40% on medium ones), while only partially improving pooled-space separation. Treating pooled cosine as a recall interface, we then benchmark verifiers scoring token--token cosine maps. MaxSim (late interaction) excels at reranking but fails to reject structural near-misses, whereas a small Transformer over similarity maps reliably separates near-misses under end-to-end training.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 20 references that are in the map. First posted on arXiv 16 Mar 2026.
Builds on
15 of 20 references.
References · 15
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2017Billion-scale similarity search with GPUscited by 2,477
- 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
- 2019Passage Re-ranking with BERTcited by 629
- 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
- 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
- 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
- 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
- 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
- 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
3 more
- 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
- 2025Vision-Language Models Do Not Understand Negationcited by 69
- 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 20
Built on by
Not cited yet.