mlsanityData to 14 Oct 2026

Bringing order into the realm of Transformer-based language models for artificial intelligence and law

arXiv 2308.05502 ↗First posted 10 Aug 2023cs.CL cs.AI cs.IRcited by 13

Not on the map. Why?

Abstract (arXiv)

Transformer-based language models (TLMs) have widely been recognized to be a cutting-edge technology for the successful development of deep-learning-based solutions to problems and applications that require natural language processing and understanding. Like for other textual domains, TLMs have indeed pushed the state-of-the-art of AI approaches for many tasks of interest in the legal domain. Despite the first Transformer model being proposed about six years ago, there has been a rapid progress of this technology at an unprecedented rate, whereby BERT and related models represent a major reference, also in the legal domain. This article provides the first systematic overview of TLM-based methods for AI-driven problems and tasks in the legal sphere. A major goal is to highlight research advances in this field so as to understand, on the one hand, how the Transformers have contributed to the success of AI in supporting legal processes, and on the other hand, what are the current limitations and opportunities for further research development.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 78 of its 137 references that are in the map. First posted on arXiv 10 Aug 2023.

Builds on

78 of 137 references.

References · 78

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  9. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 881
  10. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  11. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
38 more
  1. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  2. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  3. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  4. 2019CamemBERT: a Tasty French Language Modelcited by 368
  5. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  6. 2020MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embeddingcited by 304
  7. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  8. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 266
  9. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 246
  10. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  11. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  12. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  13. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  14. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 148
  15. 2020German's Next Language Modelcited by 144
  16. 2020How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligencecited by 129
  17. 2017A Deep Relevance Matching Model for Ad-hoc Retrievalcited by 126
  18. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 116
  19. 2020LEGAL-BERT: The Muppets straight out of Law Schoolcited by 111
  20. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 90
  21. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  22. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 87
  23. 2021ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanationcited by 78
  24. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  25. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  26. 2018CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Servicecited by 73
  27. 2023Can GPT-3 Perform Statutory Reasoning?cited by 73
  28. 2021PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarizationcited by 62
  29. 2021Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmarkcited by 60
  30. 2022FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processingcited by 49
  31. 2020The birth of Romanian BERTcited by 44
  32. 2020An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labelscited by 44
  33. 2021A Statutory Article Retrieval Dataset in Frenchcited by 40
  34. 2022EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domaincited by 40
  35. 2020GREEK-BERT: The Greeks visiting Sesame Streetcited by 38
  36. 2020PolicyQA: A Reading Comprehension Dataset for Privacy Policiescited by 34
  37. 2020Privacy Policies over Time: Curation and Analysis of a Million-Document Datasetcited by 28
  38. 2021On the Ethical Limits of Natural Language Processing on Legal Textcited by 26

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 5; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advicecited by 63
  2. 2024How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensionscited by 52
  3. 2024It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Humancited by 5
  4. 2023Justifiable Artificial Intelligence: Engineering Large Language Models for Legal Applicationscited by 2
  5. 2025TransLaw: A Large-Scale Dataset and Multi-Agent Benchmark Simulating Professional Translation of Hong Kong Case Lawcited by 2
  6. 2025Predicting potentially abusive clauses in Chilean terms of services with natural language processingcited by 1
  7. 2025Towards Mitigation of Hallucination for LLM-empowered Agents: Progressive Generalization Bound Exploration and Watchdog Monitorcited by 1
  8. 2025JustEva: A Toolkit to Evaluate LLM Fairness in Legal Knowledge Inferencecited by 1
  9. 2024Evaluating GPT-3.5's Awareness and Summarization Abilities for European Constitutional Texts with Shared Topicscited by 0
  10. 2025Retrieving Semantically Similar Decisions under Noisy Institutional Labels: Robust Comparison of Embedding Methodscited by 0
  11. 2026A Two-Stage Architecture for NDA Analysis: LLM-based Segmentation and Transformer-based Clause Classificationcited by 0
  12. 2026LegalMidm: Use-Case-Driven Legal Domain Specialization for Korean Large Language Modelcited by 0
1 more
  1. 2026Retrieval-Augmented Detection of Potentially Abusive Clauses in Chilean Terms of Servicecited by 0