mlsanityData to 14 Oct 2026

Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents

arXiv 2105.03887 ↗First posted 9 May 2021cs.CLcited by 116

Not on the map. Why?

Abstract (arXiv)

Legal artificial intelligence (LegalAI) aims to benefit legal systems with the technology of artificial intelligence, especially natural language processing (NLP). Recently, inspired by the success of pre-trained language models (PLMs) in the generic domain, many LegalAI researchers devote their effort to apply PLMs to legal tasks. However, utilizing PLMs to address legal tasks is still challenging, as the legal documents usually consist of thousands of tokens, which is far longer than the length that mainstream PLMs can process. In this paper, we release the Longformer-based pre-trained language model, named as Lawformer, for Chinese legal long documents understanding. We evaluate Lawformer on a variety of LegalAI tasks, including judgment prediction, similar case retrieval, legal reading comprehension, and legal question answering. The experimental results demonstrate that our model can achieve promising improvement on tasks with long documents as inputs.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 9 May 2021.

Builds on

28 of 42 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2018Deep contextualized word representationscited by 2,290
  8. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
16 more
  1. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  2. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  3. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
  4. 2019Neural Legal Judgment Prediction in Englishcited by 209
  5. 2018CAIL2018: A Large-Scale Legal Dataset for Judgment Predictioncited by 170
  6. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  7. 2019Adaptive Attention Span in Transformerscited by 149
  8. 2020How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligencecited by 129
  9. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 116
  10. 2020LEGAL-BERT: The Muppets straight out of Law Schoolcited by 111
  11. 2017Learning to Predict Charges for Criminal Cases with Legal Basiscited by 61
  12. 2019Legal Judgment Prediction via Multi-Perspective Bi-Feedback Networkcited by 50
  13. 2018Interpretable Charge Predictions for Criminal Cases: Learning to Generate Court Views from Fact Descriptionscited by 36
  14. 2019CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehensioncited by 26
  15. 2019BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understandingcited by 16
  16. 2021Customizing Contextualized Language Models forLegal Document Reviewscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 13; 2023: 20; 2024: 40; 2025: 24; 2026: 18.

116 papers.

Sort:

By area:

  1. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  3. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
  4. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  5. 2023Large Language Models in Law: A Surveycited by 128
  6. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  7. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 101
  8. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  9. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  10. 2024(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advicecited by 63
  11. 2023SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrievalcited by 62
  12. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 58
38 more
  1. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  2. 2023Lawyer LLaMA Technical Reportcited by 54
  3. 2023A Short Survey of Viewing Large Language Models in Legal Aspectcited by 52
  4. 2022A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challengescited by 50
  5. 2024MoralBench: Moral Evaluation of LLMscited by 50
  6. 2022Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularitiescited by 47
  7. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  8. 2024AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentationcited by 40
  9. 2022Pre-trained Language Models for the Legal Domain: A Case Study on Indian Lawcited by 39
  10. 2023LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Datasetcited by 38
  11. 2024LawLLM: Law Large Language Model for the US Legal Systemcited by 37
  12. 2022LEVEN: A Large-Scale Chinese Legal Event Detection Datasetcited by 36
  13. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  14. 2024LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Modelcited by 34
  15. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 32
  16. 2024Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?cited by 28
  17. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  18. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  19. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  20. 2022Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformercited by 20
  21. 2023Multi-Defendant Legal Judgment Prediction via Hierarchical Reasoningcited by 19
  22. 2022An Empirical Study on Cross-X Transfer for Legal Judgment Predictioncited by 16
  23. 2023THUIR@COLIEE 2023: Incorporating Structural Knowledge into Pre-trained Language Models for Legal Case Retrievalcited by 16
  24. 2024Leveraging Large Language Models for Relevance Judgments in Legal Case Retrievalcited by 16
  25. 2024Learning Interpretable Legal Case Retrieval via Knowledge-Guided Case Reformulationcited by 15
  26. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  27. 2023CaseEncoder: A Knowledge-enhanced Pre-trained Model for Legal Case Encodingcited by 14
  28. 2022BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?cited by 13
  29. 2023Bringing order into the realm of Transformer-based language models for artificial intelligence and lawcited by 13
  30. 2024NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attentioncited by 12
  31. 2024Knowledge-Infused Legal Wisdom: Navigating LLM Consultation through the Lens of Diagnostics and Positive-Unlabeled Reinforcement Learningcited by 12
  32. 2024LeKUBE: A Legal Knowledge Update BEnchmarkcited by 12
  33. 2023A Split-and-Privatize Framework for Large Language Model Fine-Tuningcited by 11
  34. 2024STARD: A Chinese Statute Retrieval Dataset with Real Queries Issued by Non-professionalscited by 10
  35. 2025CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectivescited by 10
  36. 2024DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignmentcited by 9
  37. 2025Debate-Feedback: A Multi-Agent Framework for Efficient Legal Judgment Predictioncited by 9
  38. 2022The Legal Argument Reasoning Task in Civil Procedurecited by 8

50 most cited of 116 All 116 (JSON)