mlsanityData to 14 Oct 2026

Training LLMs to be Better Text Embedders through Bidirectional Reconstruction

arXiv 2509.03020 ↗First posted 3 Sep 2025cs.CL cs.IRcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have increasingly been explored as powerful text embedders. Existing LLM-based text embedding approaches often leverage the embedding of the final token, typically a reserved special token such as [EOS]. However, these tokens have not been intentionally trained to capture the semantics of the whole context, limiting their capacity as text embeddings, especially for retrieval and re-ranking tasks. We propose to add a new training stage before contrastive learning to enrich the semantics of the final token embedding. This stage employs bidirectional generative reconstruction tasks, namely EBQ2D (Embedding-Based Query-to-Document) and EBD2Q (Embedding-Based Document-to-Query), which interleave to anchor the [EOS] embedding and reconstruct either side of Query-Document pairs. Experimental results demonstrate that our additional training stage significantly improves LLM performance on the Massive Text Embedding Benchmark (MTEB), achieving new state-of-the-art results across different LLM base models and scales.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

34 of 46 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
22 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  7. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  8. 2019ELI5: Long Form Question Answeringcited by 575
  9. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  10. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  11. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  12. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  13. 2024Generative Representational Instruction Tuningcited by 233
  14. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  15. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 119
  16. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  17. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
  18. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  19. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  20. 2023T2Ranking: A large-scale Chinese Benchmark for Passage Rankingcited by 60
  21. 2023Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Modelscited by 35
  22. 2024Conan-embedding: General Text Embedding with More and Better Negative Samplescited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Learning to Compress: Unlocking the Potential of Large Language Models for Text Representationcited by 3
  2. 2026LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden Statescited by 3
  3. 2026Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddingscited by 2
  4. 2026Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoderscited by 0
  5. 2026ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMscited by 0
  6. 2026Generative Embedding Benchmark: How Much Information Survives in a Dense Embedding?cited by 0