# Evaluating Document Representations for Content-based Legal Literature Recommendations

arXiv: https://arxiv.org/abs/2104.13841 · HTML: https://mlsanity.com/paper/2104.13841/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 57 references that are in the map.
- First posted on arXiv 28 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020)
- [A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering](https://mlsanity.com/paper/2005.05257/) (2020)
- [An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation](https://mlsanity.com/paper/1607.05368/) (2016)
- [Enriching BERT with Knowledge Graph Embeddings for Document Classification](https://mlsanity.com/paper/1909.08402/) (2019)
- [Scientific Paper Recommendation: A Survey](https://mlsanity.com/paper/2008.13538/) (2020)
- [Towards an Open Platform for Legal Information](https://mlsanity.com/paper/2005.13342/) (2020)
- [Methods for Computing Legal Document Similarity: A Comparative Study](https://mlsanity.com/paper/2004.12307/) (2020)
- [Karate Club: An API Oriented Open-source Python Framework for Unsupervised Learning on Graphs](https://mlsanity.com/paper/2003.04819/) (2020)
- [QURATOR: Innovative Technologies for Content and Data Curation](https://mlsanity.com/paper/2004.12195/) (2020)
- [Multi-Level Network Embedding with Boosted Low-Rank Matrix Approximation](https://mlsanity.com/paper/1808.08627/) (2018)

## Built on by (most cited first, 4 of 4)

- [Bringing order into the realm of Transformer-based language models for artificial intelligence and law](https://mlsanity.com/paper/2308.05502/) (2023)
- [The Law and NLP: Bridging Disciplinary Disconnects](https://mlsanity.com/paper/2310.14346/) (2023)
- [Citation Recommendation on Scholarly Legal Articles](https://mlsanity.com/paper/2311.05902/) (2023)
- [Labeling Case Similarity based on Co-Citation of Legal Articles in Judgment Documents with Empirical Dispute-Based Evaluation](https://mlsanity.com/paper/2504.20323/) (2025)

## Abstract (arXiv)

Recommender systems assist legal professionals in finding relevant literature for supporting their case. Despite its importance for the profession, legal applications do not reflect the latest advances in recommender systems and representation learning research. Simultaneously, legal recommender systems are typically evaluated in small-scale user study without any public available benchmark datasets. Thus, these studies have limited reproducibility. To address the gap between research and practice, we explore a set of state-of-the-art document representation methods for the task of retrieving semantically related US case law. We evaluate text-based (e.g., fastText, Transformers), citation-based (e.g., DeepWalk, Poincar'e), and hybrid methods. We compare in total 27 methods using two silver standards with annotations for 2,964 documents. The silver standards are newly created from Open Case Book and Wikisource and can be reused under an open license facilitating reproducibility. Our experiments show that document representations from averaged fastText word vectors (trained on legal corpora) yield the best results, closely followed by Poincar'e citation embeddings. Combining fastText and Poincar'e in a hybrid manner further improves the overall result. Besides the overall performance, we analyze the methods depending on document length, citation count, and the coverage of their recommendations. We make our source code, models, and datasets publicly available at https://github.com/malteos/legal-document-similarity/.