mlsanityData to 14 Oct 2026

Legal Evalutions and Challenges of Large Language Models

arXiv 2411.10137 ↗First posted 15 Nov 2024cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we review legal testing methods based on Large Language Models (LLMs), using the OPENAI o1 model as a case study to evaluate the performance of large models in applying legal provisions. We compare current state-of-the-art LLMs, including open-source, closed-source, and legal-specific models trained specifically for the legal domain. Systematic tests are conducted on English and Chinese legal cases, and the results are analyzed in depth. Through systematic testing of legal cases from common law systems and China, this paper explores the strengths and weaknesses of LLMs in understanding and applying legal texts, reasoning through legal issues, and predicting judgments. The experimental results highlight both the potential and limitations of LLMs in legal applications, particularly in terms of challenges related to the interpretation of legal language and the accuracy of legal reasoning. Finally, the paper provides a comprehensive analysis of the advantages and disadvantages of various types of models, offering valuable insights and references for the future application of AI in the legal field.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

32 of 52 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023Mistral 7Bcited by 3,164
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  11. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  12. 2024Yi: Open Foundation Models by 01.AIcited by 719
20 more
  1. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  2. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  3. 2023Large Language Models in Law: A Surveycited by 128
  4. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 120
  5. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  6. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  7. 2023How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?cited by 43
  8. 2023Explaining Legal Concepts with Augmented Large Language Models (GPT-4)cited by 40
  9. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  10. 2024LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Modelcited by 34
  11. 2023ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmarkcited by 33
  12. 2018LexNLP: Natural language processing and information extraction for legal and regulatory textscited by 28
  13. 2023Multimodality of AI for Education: Towards Artificial General Intelligencecited by 21
  14. 2024Assessing Large Language Models in Mechanical Engineering Education: A Study on Mechanics-Focused Conceptual Understandingcited by 15
  15. 2023LexGPT 0.1: pre-trained GPT-J models with Pile of Lawcited by 9
  16. 2024LLMs for Coding and Robotics Educationcited by 9
  17. 2024ECHOPulse: ECG controlled echocardio-grams video generationcited by 9
  18. 2023Garbage in, garbage out: Zero-shot detection of crime using Large Language Modelscited by 5
  19. 2024InternLM-Law: An Open Source Chinese Legal Large Language Modelcited by 5
  20. 2024EG-SpikeFormer: Eye-Gaze Guided Transformer on Spiking Neural Networks for Medical Image Analysiscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 27
  2. 2025Large Language Models for Bioinformaticscited by 3
  3. 2025LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Textcited by 3
  4. 2025LaQual: An Automated Framework for LLM App Quality Evaluationcited by 0