mlsanityData to 14 Oct 2026

Jasper and Stella: distillation of SOTA embedding models

arXiv 2412.19048 ↗First posted 26 Dec 2024cs.IRcited by 112

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

A crucial component in many deep learning applications, such as Frequently Asked Questions (FAQ) and Retrieval-Augmented Generation (RAG), is dense retrieval. In this process, embedding models transform raw text into numerical vectors. However, the embedding models that currently excel on text embedding benchmarks, like the Massive Text Embedding Benchmark (MTEB), often have numerous parameters and high vector dimensionality. This poses challenges for their application in real-world scenarios. To address this issue, we propose a novel multi-stage distillation framework that enables a smaller student embedding model to distill multiple larger teacher embedding models through three carefully designed losses. Meanwhile, we utilize Matryoshka Representation Learning (MRL) to reduce the vector dimensionality of the student embedding model effectively. Our student model named Jasper with 2 billion parameters, built upon the Stella embedding model, obtained the No.3 position on the MTEB leaderboard (as of December 24, 2024), achieving an average 71.54 score across 56 datasets. We have released the model and data on the Hugging Face Hub (https://huggingface.co/infgrad/jasper_en_vision_language_v1) (https://huggingface.co/datasets/infgrad/jasper_text_distill_dataset), and the training codes are available in this project repository (https://github.com/NLPJCL/RAG-Retrieval).

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. First posted on arXiv 26 Dec 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 71; 2026: 40.

112 papers.

Sort:

By area:

  1. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 106
  2. 2025Harnessing the Universal Geometry of Embeddingscited by 55
  3. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
  4. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 38
  5. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 36
  6. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  7. 2025Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-rankingcited by 31
  8. 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 28
  9. 2025Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approachescited by 23
  10. 2026jina-embeddings-v5-text: Task-Targeted Embedding Distillationcited by 22
  11. 2025Toward Multi-Session Personalized Conversation: A Large-Scale Dataset and Hierarchical Tree Framework for Implicit Reasoningcited by 21
  12. 2024GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Searchcited by 20
38 more
  1. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  2. 2025TARGET: Benchmarking Table Retrieval for Generative Taskscited by 17
  3. 2025MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphscited by 16
  4. 2025Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Videocited by 16
  5. 2025ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrievalcited by 16
  6. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  7. 2025SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrievalcited by 11
  8. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  9. 2025SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compressioncited by 10
  10. 2025Local Mixtures of Experts: Essentially Free Test-Time Training via Model Mergingcited by 9
  11. 2025Semantic-guided Diverse Decoding for Large Language Modelcited by 9
  12. 2025Rescuing the Unpoisoned: Efficient Defense against Knowledge Corruption Attacks on RAG Systemscited by 9
  13. 2026LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrievalcited by 9
  14. 2025RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systemscited by 8
  15. 2025CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detectioncited by 7
  16. 2025HUME: Measuring the Human-Model Performance Gap in Text Embedding Taskscited by 7
  17. 2025TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detectioncited by 6
  18. 2025From Pixels to Graphs: using Scene and Knowledge Graphs for HD-EPIC VQA Challengecited by 6
  19. 2025T-GRAG: A Dynamic GraphRAG Framework for Resolving Temporal Conflicts and Redundancy in Knowledge Retrievalcited by 6
  20. 2026CSRv2: Unlocking Ultra-Sparse Embeddingscited by 6
  21. 2025DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimizationcited by 5
  22. 2025Exploiting Leaderboards for Large-Scale Distribution of Malicious Modelscited by 5
  23. 2025LexSemBridge: Fine-Grained Dense Representation Enhancement through Token-Aware Embedding Augmentationcited by 5
  24. 2025Beyond Single Embeddings: Capturing Diverse Targets with Multi-Query Retrievalcited by 5
  25. 2025Fact2Fiction: Targeted Poisoning Attack to Agentic Fact-checking Systemcited by 4
  26. 2025mini-vec2vec: Scaling Universal Geometry Alignment with Linear Transformationscited by 4
  27. 2025ARK: Answer-Centric Retriever Tuning via KG-augmented Curriculum Learningcited by 4
  28. 2025Task-Oriented Automatic Fact-Checking with Frame-Semanticscited by 3
  29. 2025VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAGcited by 3
  30. 2025CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programmingcited by 3
  31. 2025QZhou-Embedding Technical Reportcited by 3
  32. 2026Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomiescited by 3
  33. 2026CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matchingcited by 3
  34. 2026CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-trainingcited by 3
  35. 2026The Consistency Illusion: How Multi-Agent Debate Hides Reasoning Misalignmentcited by 3
  36. 2025OrdRankBen: A Novel Ranking Benchmark for Ordinal Relevance in NLPcited by 2
  37. 2025Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data?cited by 2
  38. 2025Do We Really Need Specialization? Evaluating Generalist Text Embeddings for Zero-Shot Recommendation and Searchcited by 2

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
  2. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark
  3. 2025LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations
  4. 2025Dewey Long Context Embedding Model: A Technical Report
  5. 2024Linear Projections of Teacher Embeddings for Few-Class Distillation
  6. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
  7. 2026Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings
  8. 2024Mafin: Enhancing Black-Box Embeddings with Model Augmented Fine-Tuning