mlsanityData to 7 Oct 2026

Promises and pitfalls of artificial intelligence for legal applications

arXiv 2402.01656 ↗First posted 10 Jan 2024cs.CY cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Is AI set to redefine the legal profession? We argue that this claim is not supported by the current evidence. We dive into AI's increasingly prevalent roles in three types of legal tasks: information processing; tasks involving creativity, reasoning, or judgment; and predictions about the future. We find that the ease of evaluating legal applications varies greatly across legal tasks, based on the ease of identifying correct answers and the observability of information relevant to the task at hand. Tasks that would lead to the most significant changes to the legal profession are also the ones most prone to overoptimism about AI capabilities, as they are harder to evaluate. We make recommendations for better evaluation and deployment of AI in legal contexts.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 14 of its 58 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

14 of 58 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  7. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  8. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  9. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  10. 2023How is ChatGPT's behavior changing over time?cited by 265
  11. 2022Data Contamination: From Memorization to Exploitationcited by 234
  12. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 98
2 more
  1. 2020The Problem with Metrics is a Fundamental Problem for AIcited by 37
  2. 2023ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmarkcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 6; 2026: 1.

14 papers.

Sort:

By area:

  1. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  2. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  3. 2024On the Societal Impact of Open Foundation Modelscited by 64
  4. 2024(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advicecited by 61
  5. 2024Robots in the Middle: Evaluating LLMs in Dispute Resolutioncited by 13
  6. 2025We Should Evaluate Real-World Impactcited by 12
  7. 2024Lawma: The Power of Specialization for Legal Annotationcited by 7
  8. 2024CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinionscited by 5
  9. 2025Tasks and Roles in Legal AI: Data Curation, Annotation, and Verificationcited by 3
  10. 2025Bye-bye, Bluebook? Automating Legal Drudgery With AI-Augmented Rule Followingcited by 3
  11. 2025Modeling Motivated Reasoning in Law: Evaluating Strategic Role Conditioning in LLM Summarizationcited by 3
  12. 2025FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fictioncited by 1
2 more
  1. 2026Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecastingcited by 1
  2. 2025JUDGEBERT: Assessing Legal Meaning Preservation Between Sentencescited by 0