# Attentive Deep Neural Networks for Legal Document Retrieval

arXiv: https://arxiv.org/abs/2212.13899 · HTML: https://mlsanity.com/paper/2212.13899/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 55 references that are in the map.
- First posted on arXiv 13 Dec 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval](https://mlsanity.com/paper/1502.06922/) (2015)
- [JNLP Team: Deep Learning for Legal Processing in COLIEE 2020](https://mlsanity.com/paper/2011.08071/) (2020)
- [Legal Document Retrieval using Document Vector Embeddings and Deep Learning](https://mlsanity.com/paper/1805.10685/) (2018)
- [A Summary of the ALQAC 2021 Competition](https://mlsanity.com/paper/2204.10717/) (2022)
- [JNLP Team: Deep Learning Approaches for Legal Processing Tasks in COLIEE 2021](https://mlsanity.com/paper/2106.13405/) (2021)
- [ParaLaw Nets -- Cross-lingual Sentence-level Pretraining for Legal Text Processing](https://mlsanity.com/paper/2106.13403/) (2021)

## Built on by (most cited first, 13 of 13)

- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [A negation detection assessment of GPTs: analysis with the xNot360 dataset](https://mlsanity.com/paper/2306.16638/) (2023)
- [NOWJ at COLIEE 2023 -- Multi-Task and Ensemble Approaches in Legal Information Processing](https://mlsanity.com/paper/2306.04903/) (2023)
- [VLQA: The First Comprehensive, Large, and High-Quality Vietnamese Dataset for Legal Question Answering](https://mlsanity.com/paper/2507.19995/) (2025)
- [Enhancing Legal Document Retrieval: A Multi-Phase Approach with Large Language Models](https://mlsanity.com/paper/2403.18093/) (2024)
- [When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance](https://mlsanity.com/paper/2507.07748/) (2025)
- [Improving Vietnamese Legal Question--Answering System based on Automatic Data Enrichment](https://mlsanity.com/paper/2306.04841/) (2023)
- [Constructing a Knowledge Graph for Vietnamese Legal Cases with Heterogeneous Graphs](https://mlsanity.com/paper/2309.09069/) (2023)
- [RMDM: A Multilabel Fakenews Dataset for Vietnamese Evidence Verification](https://mlsanity.com/paper/2309.09071/) (2023)
- [JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledge](https://mlsanity.com/paper/2511.22869/) (2025)
- [A Deep Learning-Based System for Automatic Case Summarization](https://mlsanity.com/paper/2312.07824/) (2023)
- [Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Mining](https://mlsanity.com/paper/2507.14619/) (2025)
- [Generative Chinese Statute Retrieval](https://mlsanity.com/paper/2607.11109/) (2026)

## Abstract (arXiv)

Legal text retrieval serves as a key component in a wide range of legal text processing tasks such as legal question answering, legal case entailment, and statute law retrieval. The performance of legal text retrieval depends, to a large extent, on the representation of text, both query and legal documents. Based on good representations, a legal text retrieval model can effectively match the query to its relevant documents. Because legal documents often contain long articles and only some parts are relevant to queries, it is quite a challenge for existing models to represent such documents. In this paper, we study the use of attentive neural network-based text representation for statute law document retrieval. We propose a general approach using deep neural networks with attention mechanisms. Based on it, we develop two hierarchical architectures with sparse attention to represent long sentences and articles, and we name them Attentive CNN and Paraformer. The methods are evaluated on datasets of different sizes and characteristics in English, Japanese, and Vietnamese. Experimental results show that: i) Attentive neural methods substantially outperform non-neural methods in terms of retrieval performance across datasets and languages; ii) Pretrained transformer-based models achieve better accuracy on small datasets at the cost of high computational complexity while lighter weight Attentive CNN achieves better accuracy on large datasets; and iii) Our proposed Paraformer outperforms state-of-the-art methods on COLIEE dataset, achieving the highest recall and F2 scores in the top-N retrieval task.