mlsanityData to 7 Oct 2026

Repetition Improves Language Model Embeddings

arXiv 2402.15449 ↗First posted 23 Feb 2024cs.CL cs.LGcited by 70

Not on the map. Why?

Abstract (arXiv)

Bidirectional models are considered essential for strong text embeddings. Recent approaches to adapt autoregressive language models (LMs) into strong text embedding models have largely had the requirement to modify the LM architecture to be bidirectional. We challenge this premise by introducing "echo embeddings" which converts autoregressive LMs into high quality text embedding models without changing the architecture or requiring fine-tuning. By repeating the input and extracting embeddings from the repeated tokens -- which have access to all original tokens -- echo embeddings improve over classical LM embeddings by over 5% in zero-shot settings. Our zero-shot embeddings nearly match those obtained by bidirectionally-converted LMs that undergo additional masked-language modeling training. Echo embeddings are also compatible with supervised fine-tuning, matching or outperforming bidirectionally-converted LMs in an apples-to-apples comparison, even with an identical compute budget during training and inference. Overall, repetition is a simple and effective strategy to circumvent the need for bidirectional attention in embedding models, paving the way towards a unified architecture for all NLP tasks.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 42 of its 71 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

42 of 71 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2023Mistral 7Bcited by 3,065
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
30 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  4. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  6. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  7. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  8. 2014Distributed Representations of Sentences and Documentscited by 677
  9. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  10. 2019ELI5: Long Form Question Answeringcited by 567
  11. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  12. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  13. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  14. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  15. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  16. 2015Skip-Thought Vectorscited by 321
  17. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  18. 2023Improving Text Embeddings with Large Language Modelscited by 295
  19. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  20. 2024Generative Representational Instruction Tuningcited by 227
  21. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  22. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  23. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  24. 2023AnglE-optimized Text Embeddingscited by 98
  25. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  26. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  27. 2015Towards Universal Paraphrastic Sentence Embeddingscited by 59
  28. 2023T2Ranking: A large-scale Chinese Benchmark for Passage Rankingcited by 59
  29. 2022DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Enginecited by 15
  30. 2023Language Models are Universal Embedderscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 30; 2026: 22.

70 papers.

Sort:

By area:

  1. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  2. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  3. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  4. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  5. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  6. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  7. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  8. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  9. 2025Large Language Models are Powerful Electronic Health Record Encoderscited by 25
  10. 2025Prompt Repetition Improves Non-Reasoning LLMscited by 23
  11. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  12. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
38 more
  1. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  2. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  3. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 19
  4. 2025Should We Still Pretrain Encoders with Masked Language Modeling?cited by 15
  5. 2024OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMscited by 14
  6. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  7. 2024Crafting Interpretable Embeddings by Asking LLMs Questionscited by 11
  8. 2025A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generationcited by 10
  9. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  10. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  11. 2025Repetitions are not all alike: distinct mechanisms sustain repetition in language modelscited by 9
  12. 2025GEM: Empowering LLM for both Embedding Generation and Language Understandingcited by 8
  13. 2025F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Datacited by 8
  14. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 7
  15. 2025GRACE: Generative Representation Learning via Contrastive Policy Optimizationcited by 7
  16. 2024Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Contextcited by 6
  17. 2025HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Modelcited by 6
  18. 2025Non-Markovian Discrete Diffusion with Causal Language Modelscited by 5
  19. 2026LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculumcited by 5
  20. 2024Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learningcited by 4
  21. 2025Uncovering Gaps in How Humans and LLMs Interpret Subjective Languagecited by 4
  22. 2025LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representationscited by 4
  23. 2026BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMscited by 4
  24. 2026On Test-Time Scaling for Vision-Language Modelscited by 4
  25. 2026The Embedder's Dilemma: LLMs Are Better, but at What Cost?cited by 4
  26. 2025QZhou-Embedding Technical Reportcited by 3
  27. 2026Automatic Layer Selection for Hallucination Detectioncited by 3
  28. 2024FltLM: An Intergrated Long-Context Large Language Model for Effective Context Filtering and Understandingcited by 2
  29. 2024Better Language Models Exhibit Higher Visual Alignmentcited by 2
  30. 2024RARe: Retrieval Augmented Retrieval with In-Context Examplescited by 2
  31. 2025ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Modelscited by 2
  32. 2025Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrievalcited by 2
  33. 2026LLM2Vec-Gen: Generative Embeddings from Large Language Modelscited by 2
  34. 2026The Truth Lies Somewhere in the Middle (of the Generated Tokens)cited by 2
  35. 2026UEmbed: Unified Sparse and Dense Multimodal Embeddingscited by 2
  36. 2025A Comparative Study of Specialized LLMs as Dense Retrieverscited by 1
  37. 2025Context-Adaptive Multi-Prompt Embedding with Large Language Models for Vision-Language Alignmentcited by 1
  38. 2025BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillationcited by 1

50 most cited of 70 All 70 (JSON)