# EmbeddingGemma: Powerful and Lightweight Text Representations

arXiv: https://arxiv.org/abs/2509.20354 · HTML: https://mlsanity.com/paper/2509.20354/

## Key facts

- Cited by 139 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 32 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 28 of 28)

- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [jina-embeddings-v3: Multilingual Embeddings With Task LoRA](https://mlsanity.com/paper/2409.10173/) (2024)
- [E5-V: Universal Embeddings with Multimodal Large Language Models](https://mlsanity.com/paper/2407.12580/) (2024)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [XOR QA: Cross-lingual Open-Retrieval Question Answering](https://mlsanity.com/paper/2010.11856/) (2020)
- [Gemini Embedding: Generalizable Embeddings from Gemini](https://mlsanity.com/paper/2503.07891/) (2025)
- [NV-Retriever: Improving text embedding models with effective hard-negative mining](https://mlsanity.com/paper/2407.15831/) (2024)
- [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://mlsanity.com/paper/2403.20327/) (2024)
- [LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning](https://mlsanity.com/paper/2503.04812/) (2025)
- [LongEmbed: Extending Embedding Models for Long Context Retrieval](https://mlsanity.com/paper/2404.12096/) (2024)
- [Learning Spread-out Local Feature Descriptors](https://mlsanity.com/paper/1708.06320/) (2017)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation](https://mlsanity.com/paper/2504.06225/) (2025)
- [Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks](https://mlsanity.com/paper/2503.02656/) (2025)

## Built on by (most cited first, 30 of 139)

- [VL-JEPA: Joint Embedding Predictive Architecture for Vision-language](https://mlsanity.com/paper/2512.10942/) (2025)
- [RzenEmbed: Towards Comprehensive Multimodal Retrieval](https://mlsanity.com/paper/2510.27350/) (2025)
- [T5Gemma 2: Seeing, Reading, and Understanding Longer](https://mlsanity.com/paper/2512.14856/) (2025)
- [ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval](https://mlsanity.com/paper/2510.08252/) (2025)
- [Action100M: A Large-scale Video Action Dataset](https://mlsanity.com/paper/2601.10592/) (2026)
- [Reasoning Models Generate Societies of Thought](https://mlsanity.com/paper/2601.10825/) (2026)
- [F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World](https://mlsanity.com/paper/2603.19223/) (2026)
- [Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playing](https://mlsanity.com/paper/2601.04575/) (2026)
- [MindEval: Benchmarking Language Models on Multi-turn Mental Health Support](https://mlsanity.com/paper/2511.18491/) (2025)
- [SciCoQA: Quality Assurance for Scientific Paper--Code Alignment](https://mlsanity.com/paper/2601.12910/) (2026)
- [Diffusion-Pretrained Dense and Contextual Embeddings](https://mlsanity.com/paper/2602.11151/) (2026)
- [Semantic Search over 9 Million Mathematical Theorems](https://mlsanity.com/paper/2602.05216/) (2026)
- [Mnemis: Dual-Route Retrieval on Hierarchical Graphs for Long-Term LLM Memory](https://mlsanity.com/paper/2602.15313/) (2026)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search](https://mlsanity.com/paper/2607.27178/) (2026)
- [JEF-Hinter: Leveraging Offline Knowledge for Improving Web Agents Adaptation](https://mlsanity.com/paper/2510.04373/) (2025)
- [Search-R3: Unifying Reasoning and Embedding in Large Language Models](https://mlsanity.com/paper/2510.07048/) (2025)
- [Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Dataset](https://mlsanity.com/paper/2603.23607/) (2026)
- [PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs](https://mlsanity.com/paper/2510.06730/) (2025)
- [AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages](https://mlsanity.com/paper/2510.23896/) (2025)
- [C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling](https://mlsanity.com/paper/2512.21332/) (2025)
- [Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction](https://mlsanity.com/paper/2602.22752/) (2026)
- [Efficient and High-Fidelity Omni Modality Retrieval](https://mlsanity.com/paper/2603.02098/) (2026)
- [MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning](https://mlsanity.com/paper/2603.03379/) (2026)
- [Do LLMs Know Tool Irrelevance? Demystifying Structural Alignment Bias in Tool Invocations](https://mlsanity.com/paper/2604.11322/) (2026)
- [Is Dimensionality a Barrier for Retrieval Models?](https://mlsanity.com/paper/2605.23556/) (2026)
- [OntoLearner: A Modular Python Library for Ontology Learning with Large Language Models](https://mlsanity.com/paper/2607.01977/) (2026)
- [Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain](https://mlsanity.com/paper/2601.16018/) (2026)
- [An Extreme Multi-label Text Classification (XMTC) Library Dataset: What if we took "Use of Practical AI in Digital Libraries" seriously?](https://mlsanity.com/paper/2603.10876/) (2026)
- [Zero-Shot Quantization via Weight-Space Arithmetic](https://mlsanity.com/paper/2604.03420/) (2026)

## Abstract (arXiv)

We introduce EmbeddingGemma, a new lightweight, open text embedding model based on the Gemma 3 language model family. Our innovative training recipe strategically captures knowledge from larger models via encoder-decoder initialization and geometric embedding distillation. We improve model robustness and expressiveness with a spread-out regularizer, and ensure generalizability by merging checkpoints from varied, optimized mixtures. Evaluated on the Massive Text Embedding Benchmark (MTEB) across multilingual, English, and code domains, EmbeddingGemma (300M) achieves state-of-the-art results. Notably, it outperforms prior top models, both proprietary and open, with fewer than 500M parameters, and provides performance comparable to models double its size, offering an exceptional performance-to-cost ratio. Remarkably, this lead persists when quantizing model weights or truncating embedding outputs. This makes EmbeddingGemma particularly well-suited for low-latency and high-throughput use cases such as on-device applications. We provide ablation studies exploring our key design choices. We release EmbeddingGemma to the community to promote further research.