mlsanityData to 7 Oct 2026

The Factuality of Large Language Models in the Legal Domain

arXiv 2409.11798 ↗First posted 18 Sep 2024cs.CL cs.AI cs.IRcited by 10

Not on the map. Why?

Abstract (arXiv)

This paper investigates the factuality of large language models (LLMs) as knowledge bases in the legal domain, in a realistic usage scenario: we allow for acceptable variations in the answer, and let the model abstain from answering when uncertain. First, we design a dataset of diverse factual questions about case law and legislation. We then use the dataset to evaluate several LLMs under different evaluation methods, including exact, alias, and fuzzy matching. Our results show that the performance improves significantly under the alias and fuzzy matching methods. Further, we explore the impact of abstaining and in-context examples, finding that both strategies enhance precision. Finally, we demonstrate that additional pre-training on legal documents, as seen with SaulLM, further improves factual precision from 63% to 81%.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

34 of 48 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  12. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
22 more
  1. 2019How Can We Know What Language Models Know?cited by 953
  2. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  3. 2023Larger language models do in-context learning differentlycited by 386
  4. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  5. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  6. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  7. 2022Prompting GPT-3 To Be Reliablecited by 258
  8. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  9. 2022A Review on Language Models as Knowledge Basescited by 156
  10. 2024Long-form factuality in large language modelscited by 146
  11. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  12. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
  13. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  14. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  15. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  16. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  17. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 61
  18. 2020Machine Knowledge: Creation and Curation of Comprehensive Knowledge Basescited by 55
  19. 2023YAGO 4.5: A Large and Clean Knowledge Base with a Rich Taxonomycited by 37
  20. 2022COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Modelscited by 15
  21. 2021How to Query Language Models?cited by 11
  22. 2020Performance in the Courtroom: Automated Processing and Visualization of Appeal Court Decisions in Francecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 3.

10 papers.

  1. 2025Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answercited by 26
  2. 2025CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generationcited by 6
  3. 2025HACK: Hallucinations Along Certainty and Knowledge Axescited by 6
  4. 2025Conditioning Large Language Models on Legal Systems? Detecting Punishable Hate Speechcited by 2
  5. 2026Parthenon Law: A Self-Evolving Legal-Agent Frameworkcited by 1
  6. 2025Using Large Language Models for Legal Decision-Making in Austrian Value-Added Tax Law: A Comparative Studycited by 0
  7. 2025Retrieval-Constrained Decoding Reveals Underestimated Parametric Knowledge in Language Modelscited by 0
  8. 2025Assessing Automated Fact-Checking for Medical LLM Responses with Knowledge Graphscited by 0
  9. 2026LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMscited by 0
  10. 2026Maat: The Agentic Legal Research Assistant for Competition Protectioncited by 0