# Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering

arXiv: https://arxiv.org/abs/2505.12831 · HTML: https://mlsanity.com/paper/2505.12831/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 47 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [Generative Representational Instruction Tuning](https://mlsanity.com/paper/2402.09906/) (2024)
- [IMAGDressing-v1: Customizable Virtual Dressing](https://mlsanity.com/paper/2407.12705/) (2024)
- [Self-Detoxifying Language Models via Toxification Reversal](https://mlsanity.com/paper/2310.09573/) (2023)
- [Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics](https://mlsanity.com/paper/2403.01509/) (2024)
- [BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings](https://mlsanity.com/paper/2311.05296/) (2023)
- [Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models](https://mlsanity.com/paper/2404.03921/) (2024)
- [When Text Embedding Meets Large Language Model: A Comprehensive Survey](https://mlsanity.com/paper/2412.09165/) (2024)
- [Meaning Representations from Trajectories in Autoregressive Models](https://mlsanity.com/paper/2310.18348/) (2023)
- [Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs](https://mlsanity.com/paper/2412.11556/) (2024)
- [Composition-contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2307.07380/) (2023)
- [Multi-Prompting Decoder Helps Better Language Understanding](https://mlsanity.com/paper/2406.06279/) (2024)

## Built on by (most cited first, 13 of 13)

- [Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning](https://mlsanity.com/paper/2506.08354/) (2025)
- [Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization](https://mlsanity.com/paper/2506.15980/) (2025)
- [Steering When Necessary: Flexible Steering Large Language Models with Backtracking](https://mlsanity.com/paper/2508.17621/) (2025)
- [RegionMarker: A Region-Triggered Semantic Watermarking Framework for Embedding-as-a-Service Copyright Protection](https://mlsanity.com/paper/2511.13329/) (2025)
- [Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings](https://mlsanity.com/paper/2511.14868/) (2025)
- [LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States](https://mlsanity.com/paper/2602.01572/) (2026)
- [The Truth Lies Somewhere in the Middle (of the Generated Tokens)](https://mlsanity.com/paper/2605.09969/) (2026)
- [SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment](https://mlsanity.com/paper/2601.05075/) (2026)
- [Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMs](https://mlsanity.com/paper/2609.32684/) (2026)
- [Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detection](https://mlsanity.com/paper/2605.28664/) (2026)
- [DySem: Uncovering Dynamic Semantic Components of Large Language Models for Calculating Semantic Textual Similarity](https://mlsanity.com/paper/2605.29751/) (2026)
- [ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs](https://mlsanity.com/paper/2606.05858/) (2026)
- [Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings](https://mlsanity.com/paper/2609.37782/) (2026)

## Abstract (arXiv)

Extracting sentence embeddings from large language models (LLMs) is a practical direction, as it requires neither additional data nor fine-tuning. Previous studies usually focus on prompt engineering to guide LLMs to encode the core semantic information of the sentence into the embedding of the last token. However, the last token in these methods still encodes an excess of non-essential information, such as stop words, limiting its encoding capacity. To this end, we propose a Contrastive Prompting (CP) method that introduces an extra auxiliary prompt to elicit better sentence embedding. By contrasting with the auxiliary prompt, CP can steer existing prompts to encode the core semantics of the sentence, rather than non-essential information. CP is a plug-and-play inference-time intervention method that can be combined with various prompt-based methods. Extensive experiments on Semantic Textual Similarity (STS) tasks and downstream classification tasks demonstrate that our method can improve the performance of existing prompt-based methods across different LLMs. Our code will be released at https://github.com/zifengcheng/CP.