mlsanityData to 7 Oct 2026

Customizing Contextualized Language Models forLegal Document Reviews

arXiv 2102.05757 ↗First posted 10 Feb 2021cs.CL cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Inspired by the inductive transfer learning on computer vision, many efforts have been made to train contextualized language models that boost the performance of natural language processing tasks. These models are mostly trained on large general-domain corpora such as news, books, or Wikipedia.Although these pre-trained generic language models well perceive the semantic and syntactic essence of a language structure, exploiting them in a real-world domain-specific scenario still needs some practical considerations to be taken into account such as token distribution shifts, inference time, memory, and their simultaneous proficiency in multiple tasks. In this paper, we focus on the legal domain and present how different language model strained on general-domain corpora can be best customized for multiple legal document reviewing tasks. We compare their efficiencies with respect to task performances and present practical considerations.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 10 Feb 2021.

Builds on

22 of 33 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
10 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  2. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  3. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  4. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  5. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  6. 2017Exploring the Use of Text Classification in the Legal Domaincited by 31
  7. 2015LSTM-based Deep Learning Models for Non-factoid Answer Selectioncited by 17
  8. 2019BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understandingcited by 16
  9. 2019A Survey on Neural Network Language Modelscited by 12
  10. 2020Rapid Adaptation of BERT for Information Extraction on Domain-Specific Business Documentscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 0; 2023: 3; 2024: 3; 2025: 0; 2026: 1.

8 papers.

  1. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  2. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 115
  3. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
  4. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  5. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  6. 2024Ranking Large Language Models without Ground Truthcited by 7
  7. 2024A Comprehensive Framework for Reliable Legal AI: Combining Specialized Expert Systems and Adaptive Refinementcited by 1
  8. 2026A Two-Stage Architecture for NDA Analysis: LLM-based Segmentation and Transformer-based Clause Classificationcited by 0