# Resource-Efficient Adaptation of Large Language Models for Text Embeddings via Prompt Engineering and Contrastive Fine-tuning

arXiv: https://arxiv.org/abs/2507.22729 · HTML: https://mlsanity.com/paper/2507.22729/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [A Survey on Data Augmentation for Text Classification](https://mlsanity.com/paper/2107.03158/) (2021)
- [Scaling Sentence Embeddings with Large Language Models](https://mlsanity.com/paper/2307.16645/) (2023)
- [GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning](https://mlsanity.com/paper/2402.16829/) (2024)
- [Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models](https://mlsanity.com/paper/2404.03921/) (2024)
- [Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark](https://mlsanity.com/paper/2406.01607/) (2024)
- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)
- [Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings](https://mlsanity.com/paper/2305.10786/) (2023)

## Built on by (most cited first, 1 of 1)

- [RGAlign-Rec: Ranking-Guided Alignment for Latent Query Reasoning in Recommendation Systems](https://mlsanity.com/paper/2602.12968/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have become a cornerstone in Natural Language Processing (NLP), achieving impressive performance in text generation. Their token-level representations capture rich, human-aligned semantics. However, pooling these vectors into a text embedding discards crucial information. Nevertheless, many non-generative downstream tasks, such as clustering, classification, or retrieval, still depend on accurate and controllable sentence- or document-level embeddings. We explore several adaptation strategies for pre-trained, decoder-only LLMs: (i) various aggregation techniques for token embeddings, (ii) task-specific prompt engineering, and (iii) text-level augmentation via contrastive fine-tuning. Combining these components yields competitive performance on the English clustering track of the Massive Text Embedding Benchmark (MTEB). An analysis of the attention map further shows that fine-tuning shifts focus from prompt tokens to semantically relevant words, indicating more effective compression of meaning into the final hidden state. Our experiments demonstrate that LLMs can be effectively adapted as text embedding models through a combination of prompt engineering and resource-efficient contrastive fine-tuning on synthetically generated positive pairs.