mlsanityData to 7 Oct 2026

Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores

arXiv 2408.11868 ↗First posted 19 Aug 2024cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper presents an approach to improve text embedding models through contrastive fine-tuning on small datasets augmented with expert scores. It focuses on enhancing semantic textual similarity tasks and addressing text retrieval problems. The proposed method uses soft labels derived from expert-augmented scores to fine-tune embedding models, preserving their versatility and ensuring retrieval capability is improved. The paper evaluates the method using a Q&A dataset from an online shopping website and eight expert models. Results show improved performance over a benchmark model across multiple metrics on various retrieval tasks from the massive text embedding benchmark (MTEB). The method is cost-effective and practical for real-world applications, especially when labeled data is scarce.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

24 of 31 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  7. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  8. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  9. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  10. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  11. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  12. 2023Improving Text Embeddings with Large Language Modelscited by 295
12 more
  1. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  2. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  3. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  4. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  5. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  6. 2020Top2Vec: Distributed Representations of Topicscited by 124
  7. 2024Evaluating Retrieval Quality in Retrieval-Augmented Generationcited by 90
  8. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  9. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  10. 2018Online Embedding Compression for Text Classification using Low Rank Matrix Factorizationcited by 22
  11. 2023Is Anisotropy Inherent to Transformers?cited by 5
  12. 2024Low-Rank Approximation, Adaptation, and Other Talescited by 1

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systemscited by 9
  2. 2025Large Language Model Compression via the Nested Activation-Aware Decompositioncited by 0
  3. 2025Scalable Unit Harmonization in Medical Informatics via Bayesian-Optimized Retrieval and Transformer-Based Re-rankingcited by 0