# Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMs

arXiv: https://arxiv.org/abs/2609.32684 · HTML: https://mlsanity.com/paper/2609.32684/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [Goal-Driven Explainable Clustering via Language Descriptions](https://mlsanity.com/paper/2305.13749/) (2023)
- [Self-Detoxifying Language Models via Toxification Reversal](https://mlsanity.com/paper/2310.09573/) (2023)
- [Why Larger Language Models Do In-context Learning Differently?](https://mlsanity.com/paper/2405.19592/) (2024)
- [Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineering](https://mlsanity.com/paper/2410.16314/) (2024)
- [WASSA-2017 Shared Task on Emotion Intensity](https://mlsanity.com/paper/1708.03700/) (2017)
- [BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings](https://mlsanity.com/paper/2311.05296/) (2023)
- [MasakhaNEWS: News Topic Classification for African languages](https://mlsanity.com/paper/2304.09972/) (2023)
- [Answer is All You Need: Instruction-following Text Embedding via Answering the Question](https://mlsanity.com/paper/2402.09642/) (2024)
- [Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs](https://mlsanity.com/paper/2412.11556/) (2024)
- [Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering](https://mlsanity.com/paper/2505.12831/) (2025)
- [Hyper-CL: Conditioning Sentence Representations with Hypernetworks](https://mlsanity.com/paper/2403.09490/) (2024)
- [Steering When Necessary: Flexible Steering Large Language Models with Backtracking](https://mlsanity.com/paper/2508.17621/) (2025)
- [Out-of-the-Box Conditional Text Embeddings from Large Language Models](https://mlsanity.com/paper/2504.16411/) (2025)
- [RegionMarker: A Region-Triggered Semantic Watermarking Framework for Embedding-as-a-Service Copyright Protection](https://mlsanity.com/paper/2511.13329/) (2025)

## Built on by (most cited first, 1 of 1)

- [Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay](https://mlsanity.com/paper/2610.05564/) (2026)

## Abstract (arXiv)

Extracting conditional text embeddings from large language models (LLMs) is a promising paradigm, as it requires neither additional data nor fine-tuning. Existing methods incorporate conditions into prompts to guide LLMs to focus on specific aspects and elicit conditional text embeddings. However, relying solely on prompts often fails to produce high-quality conditional text embeddings, as they remain entangled with general text embeddings, ultimately degrading their quality. To this end, we propose an inference-time, plug-and-play Self-Contrastive Steering (SCS) method that constructs unconditional general text embeddings and uses them to refine conditional text embeddings, making them more focused on the target condition. Specifically, we modify the attention mask and positional encodings to mask the condition, thereby obtaining unconditional text embeddings and intervening in the multi-head self-attention computation process. Notably, our method is highly efficient, requiring only a single additional multi-head self-attention computation at inference time. Extensive experiments on clustering, Semantic Textual Similarity, and triplet alignment datasets demonstrate that our method can seamlessly improve the performance of existing prompt-based methods across different LLMs in a training-free and plug-and-play manner. Our code will be released at https://github.com/zifengcheng/SCS