# Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings

arXiv: https://arxiv.org/abs/2601.11124 · HTML: https://mlsanity.com/paper/2601.11124/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 16 Jan 2026.

## Builds on (most cited first, 22 of 22)

- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Compressed Chain of Thought: Efficient Reasoning Through Dense Representations](https://mlsanity.com/paper/2412.13171/) (2024)
- [Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training](https://mlsanity.com/paper/2306.03166/) (2023)
- [FinMTEB: Finance Massive Text Embedding Benchmark](https://mlsanity.com/paper/2502.10990/) (2025)
- [Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment](https://mlsanity.com/paper/2502.11401/) (2025)
- [ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain](https://mlsanity.com/paper/2412.00532/) (2024)
- [Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings](https://mlsanity.com/paper/2506.01435/) (2025)
- [U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs](https://mlsanity.com/paper/2507.14902/) (2025)
- [Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinement](https://mlsanity.com/paper/2509.24291/) (2025)
- [GEM: Empowering LLM for both Embedding Generation and Language Understanding](https://mlsanity.com/paper/2506.04344/) (2025)
- [GRACE: Generative Representation Learning via Contrastive Policy Optimization](https://mlsanity.com/paper/2510.04506/) (2025)
- [Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings](https://mlsanity.com/paper/2506.15001/) (2025)
- [Training LLMs to be Better Text Embedders through Bidirectional Reconstruction](https://mlsanity.com/paper/2509.03020/) (2025)
- [ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings](https://mlsanity.com/paper/2508.01643/) (2025)
- [Retrieval-GRPO: A Multi-Objective Reinforcement Learning Framework for Dense Retrieval in Taobao Search](https://mlsanity.com/paper/2511.13885/) (2025)
- [Generative Representational Learning of Foundation Models for Recommendation](https://mlsanity.com/paper/2506.11999/) (2025)
- [Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrieval](https://mlsanity.com/paper/2509.00276/) (2025)
- [TermGPT: Multi-Level Contrastive Fine-Tuning for Terminology Adaptation in Legal and Financial Domain](https://mlsanity.com/paper/2511.09854/) (2025)

## Built on by (most cited first, 2 of 2)

- [Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings](https://mlsanity.com/paper/2604.22280/) (2026)
- [AtomRec: Evolving Atomic Memory for Agentic Recommendation](https://mlsanity.com/paper/2609.04882/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) adapted via contrastive learning excel in general representation learning but struggle in vertical domains like chemistry and law, primarily due to a lack of domain-specific knowledge. This work identifies a core bottleneck: the prevailing ``LLM+CL'' paradigm focuses on semantic alignment but cannot perform knowledge acquisition, leading to failures on specialized terminology. To bridge this gap, we propose Learn Before Represent (LBR), a novel two-stage framework. LBR first injects domain knowledge via an Information Bottleneck-Constrained Generative Learning stage, preserving the LLM's causal attention to maximize knowledge acquisition while compressing semantics. It then performs Generative-Refined Contrastive Learning on the compressed representations for alignment. This approach maintains architectural consistency and resolves the objective conflict between generative and contrastive learning. Extensive experiments on medical, chemistry, and code retrieval tasks show that LBR significantly outperforms strong baselines. Our work establishes a new paradigm for building accurate and robust representations in vertical domains.