mlsanityData to 7 Oct 2026

Evaluating AI for Law: Bridging the Gap with Open-Source Solutions

arXiv 2404.12349 ↗First posted 18 Apr 2024cs.AI cs.HCcited by 11

Not on the map. Why?

Abstract (arXiv)

This study evaluates the performance of general-purpose AI, like ChatGPT, in legal question-answering tasks, highlighting significant risks to legal professionals and clients. It suggests leveraging foundational models enhanced by domain-specific knowledge to overcome these issues. The paper advocates for creating open-source legal AI systems to improve accuracy, transparency, and narrative diversity, addressing general AI's shortcomings in legal contexts.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

15 of 32 references.

References · 15

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  6. 2018World Modelscited by 1,444
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  8. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  9. 2019Discrimination in the Age of Algorithmscited by 51
  10. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergencecited by 42
  11. 2024Better Call GPT, Comparing Large Language Models Against Lawyerscited by 25
  12. 2022LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoningcited by 17
3 more
  1. 2023Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreementcited by 17
  2. 2023Prototype-Based Interpretability for Legal Citation Predictioncited by 9
  3. 2023OpenAI Cribbed Our Tax Example, But Can GPT-4 Really Do Tax?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 3.

11 papers.

  1. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  2. 2025GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citationscited by 15
  3. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  4. 2024Lawma: The Power of Specialization for Legal Annotationcited by 7
  5. 2026LLM Agents in Law: Taxonomy, Applications, and Challengescited by 5
  6. 2025Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classificationcited by 2
  7. 2025Evaluating the Role of Large Language Models in Legal Practice in Indiacited by 2
  8. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  9. 2025Does It Tie Out? Towards Autonomous Legal Agents in Venture Capitalcited by 0
  10. 2026AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Argumentscited by 0
  11. 2026From Judgments to Issues: Structured Extraction of Legal Reasoning with Citation-Hallucination Controlcited by 0