# CaseEncoder: A Knowledge-enhanced Pre-trained Model for Legal Case Encoding

arXiv: https://arxiv.org/abs/2305.05393 · HTML: https://mlsanity.com/paper/2305.05393/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://mlsanity.com/paper/2002.10857/) (2020)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval](https://mlsanity.com/paper/2010.10137/) (2020)
- [Legal Case Document Similarity: You Need Both Network and Text](https://mlsanity.com/paper/2209.12474/) (2022)
- [Learning Fine-grained Fact-Article Correspondence in Legal Cases](https://mlsanity.com/paper/2104.10726/) (2021)

## Built on by (most cited first, 14 of 14)

- [Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models](https://mlsanity.com/paper/2403.06448/) (2024)
- [DRAGIN: Dynamic Retrieval Augmented Generation based on the Information Needs of Large Language Models](https://mlsanity.com/paper/2403.10081/) (2024)
- [THUIR@COLIEE 2023: Incorporating Structural Knowledge into Pre-trained Language Models for Legal Case Retrieval](https://mlsanity.com/paper/2305.06812/) (2023)
- [SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation](https://mlsanity.com/paper/2508.15658/) (2025)
- [Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-hoc Retrieval](https://mlsanity.com/paper/2312.10661/) (2023)
- [Learning Interpretable Legal Case Retrieval via Knowledge-Guided Case Reformulation](https://mlsanity.com/paper/2406.19760/) (2024)
- [Joint Evaluation of Answer and Reasoning Consistency for Hallucination Detection in Large Reasoning Models](https://mlsanity.com/paper/2506.04832/) (2025)
- [LeKUBE: A Legal Knowledge Update BEnchmark](https://mlsanity.com/paper/2407.14192/) (2024)
- [STARD: A Chinese Statute Retrieval Dataset with Real Queries Issued by Non-professionals](https://mlsanity.com/paper/2406.15313/) (2024)
- [ELLA: Empowering LLMs for Interpretable, Accurate and Informative Legal Advice](https://mlsanity.com/paper/2408.07137/) (2024)
- [Dynamic and Parametric Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.06704/) (2025)
- [From Query to Counsel: Structured Reasoning with a Multi-Agent Framework and Dataset for Legal Consultation](https://mlsanity.com/paper/2604.10470/) (2026)
- [Segment First, Retrieve Better: Realistic Legal Search via Rhetorical Role-Based Queries](https://mlsanity.com/paper/2508.00679/) (2025)
- [NeuroSymbolic AI for Legal AI-TRISM: Trustworthy, Reliable, Interpretable, Safe Models](https://mlsanity.com/paper/2606.15646/) (2026)

## Abstract (arXiv)

Legal case retrieval is a critical process for modern legal information systems. While recent studies have utilized pre-trained language models (PLMs) based on the general domain self-supervised pre-training paradigm to build models for legal case retrieval, there are limitations in using general domain PLMs as backbones. Specifically, these models may not fully capture the underlying legal features in legal case documents. To address this issue, we propose CaseEncoder, a legal document encoder that leverages fine-grained legal knowledge in both the data sampling and pre-training phases. In the data sampling phase, we enhance the quality of the training data by utilizing fine-grained law article information to guide the selection of positive and negative examples. In the pre-training phase, we design legal-specific pre-training tasks that align with the judging criteria of relevant legal cases. Based on these tasks, we introduce an innovative loss function called Biased Circle Loss to enhance the model's ability to recognize case relevance in fine grains. Experimental results on multiple benchmarks demonstrate that CaseEncoder significantly outperforms both existing general pre-training models and legal-specific pre-training models in zero-shot legal case retrieval.