mlsanityData to 14 Oct 2026

Understanding the Influence of Synthetic Data for Text Embedders

arXiv 2509.06184 ↗First posted 7 Sep 2025cs.CLcited by 1

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Recent progress in developing general purpose text embedders has been driven by training on ever-growing corpora of synthetic LLM-generated data. Nonetheless, no publicly available synthetic dataset exists, posing a barrier to studying its role for generalization. To address this issue, we first reproduce and publicly release the synthetic data proposed by Wang et al. (Mistral-E5). Our synthetic data is high quality and leads to consistent improvements in performance. Next, we critically examine where exactly synthetic data improves model generalization. Our analysis reveals that benefits from synthetic data are sparse and highly localized to individual datasets. Moreover, we observe trade-offs between the performance on different categories and data that benefits one task, degrades performance on another. Our findings highlight the limitations of current synthetic data approaches for building general-purpose embedders and challenge the notion that training on synthetic data leads to more robust embedding models across tasks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 12 of its 19 references that are in the map. First posted on arXiv 7 Sep 2025.

Builds on

12 of 19 references.

References · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  7. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  8. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  9. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 235
  10. 2019Document Expansion by Query Predictioncited by 231
  11. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  12. 2024Little Giants: Synthesizing High-Quality Embedding Data at Scalecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  2. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark
  3. 2024Efficacy of Synthetic Data as a Benchmark
  4. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  5. 2025Scaling Laws of Synthetic Data for Language Models
  6. 2024How to Synthesize Text Data without Model Collapse?
  7. 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data
  8. 2022Reweighting Strategy based on Synthetic Data Identification for Sentence Similarity