# Meta-Task Prompting Elicits Embeddings from Large Language Models

arXiv: https://arxiv.org/abs/2402.18458 · HTML: https://mlsanity.com/paper/2402.18458/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 58 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Fine-Tuning LLaMA for Multi-Stage Text Retrieval](https://mlsanity.com/paper/2310.08319/) (2023)
- [Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT](https://mlsanity.com/paper/2302.10198/) (2023)
- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2204.10298/) (2022)
- [Task-aware Retrieval with Instructions](https://mlsanity.com/paper/2211.09260/) (2022)
- [Learning to Match Using Local and Distributed Representations of Text for Web Search](https://mlsanity.com/paper/1610.08136/) (2016)
- [HyperPrompt: Prompt-based Task-Conditioning of Transformers](https://mlsanity.com/paper/2203.00759/) (2022)
- [BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings](https://mlsanity.com/paper/2311.05296/) (2023)
- [PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings](https://mlsanity.com/paper/2201.12093/) (2022)
- [Improving Contrastive Learning of Sentence Embeddings from AI Feedback](https://mlsanity.com/paper/2305.01918/) (2023)
- [Contrastive Learning of Sentence Embeddings from Scratch](https://mlsanity.com/paper/2305.15077/) (2023)
- [Meaning Representations from Trajectories in Autoregressive Models](https://mlsanity.com/paper/2310.18348/) (2023)

## Built on by (most cited first, 16 of 16)

- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free](https://mlsanity.com/paper/2410.10814/) (2024)
- [Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models](https://mlsanity.com/paper/2404.03921/) (2024)
- [FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training](https://mlsanity.com/paper/2411.11927/) (2024)
- [Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features](https://mlsanity.com/paper/2412.00142/) (2024)
- [FreeRet: MLLMs as Training-Free Retrievers](https://mlsanity.com/paper/2509.24621/) (2025)
- [LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States](https://mlsanity.com/paper/2602.01572/) (2026)
- [LLM2Vec-Gen: Generative Embeddings from Large Language Models](https://mlsanity.com/paper/2603.10913/) (2026)
- [Context-Adaptive Multi-Prompt Embedding with Large Language Models for Vision-Language Alignment](https://mlsanity.com/paper/2508.02762/) (2025)
- [mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrieval](https://mlsanity.com/paper/2604.17054/) (2026)
- [Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings](https://mlsanity.com/paper/2606.07502/) (2026)
- [KunlunBaize: LLM with Multi-Scale Convolution and Multi-Token Prediction Under TransformerX Framework](https://mlsanity.com/paper/2503.04784/) (2025)
- [Lens: Bringing the Right Semantic Perspective into Focus for Training-Free Multimodal Representation Learning](https://mlsanity.com/paper/2609.20252/) (2026)
- [Coherence-Aware Distributional Evaluation of Open-Ended Text Generation](https://mlsanity.com/paper/2609.34240/) (2026)
- [Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings](https://mlsanity.com/paper/2609.37782/) (2026)
- [Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay](https://mlsanity.com/paper/2610.05564/) (2026)

## Abstract (arXiv)

We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation (MetaEOL), for generating high-quality sentence embeddings from Large Language Models (LLMs) without the need for model fine-tuning. Leveraging meta-task prompting, MetaEOL guides LLMs to produce embeddings through a series of carefully designed prompts that address multiple representational aspects. Our comprehensive experiments demonstrate that embeddings averaged from various meta-tasks are versatile embeddings that yield competitive performance on Semantic Textual Similarity (STS) benchmarks and excel in downstream tasks, surpassing contrastive-trained models. Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.