# DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment

arXiv: https://arxiv.org/abs/2403.18435 · HTML: https://mlsanity.com/paper/2403.18435/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 18 of 18)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://mlsanity.com/paper/2108.05540/) (2021)
- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval](https://mlsanity.com/paper/2207.02578/) (2022)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)
- [SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval](https://mlsanity.com/paper/2304.11370/) (2023)
- [T2Ranking: A large-scale Chinese Benchmark for Passage Ranking](https://mlsanity.com/paper/2304.03679/) (2023)
- [LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset](https://mlsanity.com/paper/2310.17609/) (2023)
- [Neural Machine Translation with Supervised Attention](https://mlsanity.com/paper/1609.04186/) (2016)
- [Adding Interpretable Attention to Neural Translation Models Improves Word Alignment](https://mlsanity.com/paper/1901.11359/) (2019)
- [Legal Case Document Similarity: You Need Both Network and Text](https://mlsanity.com/paper/2209.12474/) (2022)
- [THUIR@COLIEE 2023: Incorporating Structural Knowledge into Pre-trained Language Models for Legal Case Retrieval](https://mlsanity.com/paper/2305.06812/) (2023)
- [Constructing Tree-based Index for Efficient and Effective Dense Retrieval](https://mlsanity.com/paper/2304.11943/) (2023)
- [THUIR@COLIEE 2023: More Parameters and Legal Knowledge for Legal Case Entailment](https://mlsanity.com/paper/2305.06817/) (2023)
- [Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder](https://mlsanity.com/paper/2102.09206/) (2021)

## Built on by (most cited first, 9 of 9)

- [LegalAgentBench: Evaluating LLM Agents in Legal Domain](https://mlsanity.com/paper/2412.17259/) (2024)
- [LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2409.20288/) (2024)
- [LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation](https://mlsanity.com/paper/2502.20640/) (2025)
- [CaseGen: A Benchmark for Multi-Stage Legal Case Documents Generation](https://mlsanity.com/paper/2502.17943/) (2025)
- [JUREX-4E: Juridical Expert-Annotated Four-Element Knowledge Base for Legal Reasoning](https://mlsanity.com/paper/2502.17166/) (2025)
- [LLM-based Embedders for Prior Case Retrieval](https://mlsanity.com/paper/2507.18455/) (2025)
- [LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval](https://mlsanity.com/paper/2602.09570/) (2026)
- [PL-CA: A Parametric Legal Case Augmentation Framework](https://mlsanity.com/paper/2509.06356/) (2025)
- [PRecG: Legal Precedent Retrieval with Graph Neural Networks and Rhetorical Role Segmentation](https://mlsanity.com/paper/2607.09094/) (2026)

## Abstract (arXiv)

Recent research demonstrates the effectiveness of using pre-trained language models for legal case retrieval. Most of the existing works focus on improving the representation ability for the contextualized embedding of the [CLS] token and calculate relevance using textual semantic similarity. However, in the legal domain, textual semantic similarity does not always imply that the cases are relevant enough. Instead, relevance in legal cases primarily depends on the similarity of key facts that impact the final judgment. Without proper treatments, the discriminative ability of learned representations could be limited since legal cases are lengthy and contain numerous non-key facts. To this end, we introduce DELTA, a discriminative model designed for legal case retrieval. The basic idea involves pinpointing key facts in legal cases and pulling the contextualized embedding of the [CLS] token closer to the key facts while pushing away from the non-key facts, which can warm up the case embedding space in an unsupervised manner. To be specific, this study brings the word alignment mechanism to the contextual masked auto-encoder. First, we leverage shallow decoders to create information bottlenecks, aiming to enhance the representation ability. Second, we employ the deep decoder to enable translation between different structures, with the goal of pinpointing key facts to enhance discriminative ability. Comprehensive experiments conducted on publicly available legal benchmarks show that our approach can outperform existing state-of-the-art methods in legal case retrieval. It provides a new perspective on the in-depth understanding and processing of legal case documents.