# HEAL: Hierarchical Embedding Alignment Loss for Improved Retrieval and Representation Learning

arXiv: https://arxiv.org/abs/2412.04661 · HTML: https://mlsanity.com/paper/2412.04661/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 6 of 6)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [HiClass: a Python library for local hierarchical classification compatible with scikit-learn](https://mlsanity.com/paper/2112.06560/) (2021)
- [Interactive Distillation of Large Single-Topic Corpora of Scientific Papers](https://mlsanity.com/paper/2309.10772/) (2023)

## Built on by (most cited first, 3 of 3)

- [Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval](https://mlsanity.com/paper/2505.05666/) (2025)
- [SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoning](https://mlsanity.com/paper/2507.16518/) (2025)
- [Osiris: A Lightweight Open-Source Hallucination Detection System](https://mlsanity.com/paper/2505.04844/) (2025)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external document retrieval to provide domain-specific or up-to-date knowledge. The effectiveness of RAG depends on the relevance of retrieved documents, which is influenced by the semantic alignment of embeddings with the domain's specialized content. Although full fine-tuning can align language models to specific domains, it is computationally intensive and demands substantial data. This paper introduces Hierarchical Embedding Alignment Loss (HEAL), a novel method that leverages hierarchical fuzzy clustering with matrix factorization within contrastive learning to efficiently align LLM embeddings with domain-specific content. HEAL computes level/depth-wise contrastive losses and incorporates hierarchical penalties to align embeddings with the underlying relationships in label hierarchies. This approach enhances retrieval relevance and document classification, effectively reducing hallucinations in LLM outputs. In our experiments, we benchmark and evaluate HEAL across diverse domains, including Healthcare, Material Science, Cyber-security, and Applied Maths.