# KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model

arXiv: https://arxiv.org/abs/2501.01028 · HTML: https://mlsanity.com/paper/2501.01028/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- First posted on arXiv 2 Jan 2025.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Built on by (most cited first, 30 of 36)

- [KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model](https://mlsanity.com/paper/2506.20923/) (2025)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension](https://mlsanity.com/paper/2508.01959/) (2025)
- [Semantic Search over 9 Million Mathematical Theorems](https://mlsanity.com/paper/2602.05216/) (2026)
- [MultiClaimNet: A Massively Multilingual Dataset of Fact-Checked Claim Clusters](https://mlsanity.com/paper/2503.22280/) (2025)
- [LMEB: Long-horizon Memory Embedding Benchmark](https://mlsanity.com/paper/2603.12572/) (2026)
- [Exploiting Leaderboards for Large-Scale Distribution of Malicious Models](https://mlsanity.com/paper/2507.08983/) (2025)
- [MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch](https://mlsanity.com/paper/2509.12340/) (2025)
- [BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs](https://mlsanity.com/paper/2604.02045/) (2026)
- [Balancing the Blend: An Experimental Analysis of Trade-offs in Hybrid Search](https://mlsanity.com/paper/2508.01405/) (2025)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages](https://mlsanity.com/paper/2510.23896/) (2025)
- [How Do Decoder-Only LLMs Perceive Users? Rethinking Attention Masking for User Representation Learning](https://mlsanity.com/paper/2602.10622/) (2026)
- [One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation](https://mlsanity.com/paper/2605.22544/) (2026)
- [Does My Embedding Reflect That A = B? Evaluating Mathematical Equivalence in Embedding Models](https://mlsanity.com/paper/2606.23959/) (2026)
- [PolitiSky24: U.S. Political Bluesky Dataset with User Stance Labels](https://mlsanity.com/paper/2506.07606/) (2025)
- [Do We Really Need Specialization? Evaluating Generalist Text Embeddings for Zero-Shot Recommendation and Search](https://mlsanity.com/paper/2507.05006/) (2025)
- [Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions](https://mlsanity.com/paper/2509.20645/) (2025)
- [Neural network embeddings recover value dimensions from psychometric survey items on par with human data](https://mlsanity.com/paper/2509.24906/) (2025)
- [Multilingual vs Crosslingual Retrieval of Fact-Checked Claims: A Tale of Two Approaches](https://mlsanity.com/paper/2505.22118/) (2025)
- [HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation](https://mlsanity.com/paper/2602.07739/) (2026)
- [Query as Anchor: Scenario-Adaptive User Representation via Large Language Model](https://mlsanity.com/paper/2602.14492/) (2026)
- [Prism-Reranker: Beyond Relevance Scoring -- Jointly Producing Contributions and Evidence for Agentic Retrieval](https://mlsanity.com/paper/2604.23734/) (2026)
- [Causal2Vec: Improving Decoder-only LLMs as Embedding Models through a Contextual Token](https://mlsanity.com/paper/2507.23386/) (2025)
- [Semi-Supervised Synthetic Data Generation with Fine-Grained Relevance Control for Short Video Search Relevance Modeling](https://mlsanity.com/paper/2509.16717/) (2025)
- [TurkEmbed: Turkish Embedding Model on NLI & STS Tasks](https://mlsanity.com/paper/2511.08376/) (2025)
- [CQ-CiM: Hardware-Aware Embedding Shaping for Robust CiM-Based Retrieval](https://mlsanity.com/paper/2602.20083/) (2026)
- [CONE: Embeddings for Complex Numerical Data Preserving Unit and Variable Semantics](https://mlsanity.com/paper/2603.04741/) (2026)
- [Iterative Definition Refinement for Zero-Shot Classification via LLM-Based Semantic Prototype Optimization](https://mlsanity.com/paper/2604.27335/) (2026)
- [Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance](https://mlsanity.com/paper/2605.22202/) (2026)

## Abstract (arXiv)

As retrieval-augmented generation prevails in large language models, embedding models are becoming increasingly crucial. Despite the growing number of general embedding models, prior work often overlooks the critical role of training data quality. In this work, we introduce KaLM-Embedding, a general multilingual embedding model that leverages a large quantity of cleaner, more diverse, and domain-specific training data. Our model has been trained with key techniques proven to enhance performance: (1) persona-based synthetic data to create diversified examples distilled from LLMs, (2) ranking consistency filtering to remove less informative samples, and (3) semi-homogeneous task batch sampling to improve training efficacy. Departing from traditional BERT-like architectures, we adopt Qwen2-0.5B as the pre-trained model, facilitating the adaptation of auto-regressive language models for general embedding tasks. Extensive evaluations of the MTEB benchmark across multiple languages show that our model outperforms others of comparable size, setting a new standard for multilingual embedding models with <1B parameters.