Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Sampling
On the map: Retrieval-augmented generation → Generation and context integration
Abstract (arXiv)
In this paper we present APEX-Embedding-7B (Advanced Processing for Epistemic eXtraction), a 7-billion parameter decoder-only text Feature Extraction Model, specifically designed for Document Retrieval-Augmented Generation (RAG) tasks. Our approach employs two training techniques that yield an emergent improvement in factual focus: (1) Pre-convergence interrupted fine-tuning using Structured Entity Relationship Maps as training data input: designed to shift the model's attention and create a bias towards factual content rather than semantic style - this enhances plain text performance despite not being directly trained for it; and (2) Model-Aware Contrastive Sampling, creating a balanced and evenly distributed collation map of hard and soft negatives directly informed by the base model's competency. This combined methodology yields significant improvements, enhancing plain text query/document pair retrieval to achieve an absolute rank@1 accuracy of 90.86% (an increase of 6.26% compared to the next leading model) in our evaluation, and reducing training data input context size by an average of 37.71% compared to plain text for both queries and document texts. Based on our evaluations, our model establishes a new state-of-the-art standard in text feature extraction for longer context document retrieval tasks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 18 of its 21 references that are in the map. First posted on arXiv 8 Oct 2024.
Builds on
18 of 21 references.
Closest ancestors · 1
Same direction, newest first
Other references · 17
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2023Mistral 7Bcited by 3,164
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
- 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
- 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
- 2022MTEB: Massive Text Embedding Benchmarkcited by 640
- 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
- 2023Improving Text Embeddings with Large Language Modelscited by 303
- 2024The Prompt Report: A Systematic Survey of Prompt Engineering Techniquescited by 121
- 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 88
5 more
- 2017Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networkscited by 59
- 2023Emergent Representations of Program Semantics in Language Models Trained on Programscited by 30
- 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
- 2023Deliberate then Generate: Enhanced Prompting Framework for Text Generationcited by 13
- 2022Model-Aware Contrastive Learning: Towards Escaping the Dilemmascited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
- 2025Embedding-Based Context-Aware Reranker
- 2025ELITE: Embedding-Less retrieval with Iterative Text Exploration
- 2025Mixture-of-RAG: Integrating Text and Tables with Large Language Models
- 2025Learning Refined Document Representations for Dense Retrieval via Deliberate Thinking
- 2026CHOP: Chunkwise Context-Preserving Framework for RAG on Multi Documents
- 2025SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression
- 2026Mapping Similarity Spaces across Embedding Models with Synthetic Query Probing