mlsanityData to 7 Oct 2026

Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering

arXiv 2501.06521 ↗First posted 11 Jan 2025cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Hallucination, or the generation of incorrect or fabricated information, remains a critical challenge in large language models (LLMs), particularly in high-stake domains such as legal question answering (QA). In order to mitigate the hallucination rate in legal QA, we first introduce a benchmark called LegalHalBench and three automatic metrics to evaluate the common hallucinations when LLMs answer legal questions. We then propose a hallucination mitigation method that integrates behavior cloning and a novel Hard Sample-aware Iterative Direct Preference Optimization (HIPO). We conduct extensive real-data experiments to validate the effectiveness of our approach. Our results demonstrate remarkable improvements in various metrics, including the newly proposed Non-Hallucinated Statute Rate, Statute Relevance Rate, Legal Claim Truthfulness, as well as traditional metrics such as METEOR, BERTScore, ROUGE-L, and win rates.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 30 of its 55 references that are in the map. First posted on arXiv 11 Jan 2025.

Builds on

30 of 55 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2024Qwen2 Technical Reportcited by 2,076
  11. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
18 more
  1. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  2. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  3. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  4. 2023Fine-tuning Language Models for Factualitycited by 219
  5. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  6. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  7. 2018CAIL2018: A Large-Scale Legal Dataset for Judgment Predictioncited by 168
  8. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  9. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  10. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedbackcited by 80
  11. 2023Can GPT-3 Perform Statutory Reasoning?cited by 71
  12. 2023FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasetscited by 68
  13. 2022Legal Prompt Engineering for Multilingual Legal Judgement Predictioncited by 54
  14. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Taskscited by 35
  15. 2022Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Predictioncited by 8
  16. 2024Enhancing Court View Generation with Knowledge Injection and Guidancecited by 5
  17. 2021Dialogue Inspectional Summarization with Factual Inconsistency Awarenesscited by 3
  18. 2024From Graph to Word Bag: Introducing Domain Knowledge to Confusing Charge Predictioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 16.

28 papers.

Sort:

By area:

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025Towards Robust Legal Reasoning: Harnessing Logical LLMs in Lawcited by 30
  3. 2025Explainable Rule Application via Structured Prompting: A Neural-Symbolic Approachcited by 14
  4. 2025Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyondcited by 9
  5. 2026LRAS: Advanced Legal Reasoning with Agentic Searchcited by 8
  6. 2025Challenges for Generative AI in Legal Reasoningcited by 7
  7. 2025LLM Abstention Can Be a Prompt Artifact, in Addition to Genuine Uncertaintycited by 4
  8. 2025WaterMod: Modular Token-Rank Partitioning for Probability-Balanced LLM Watermarkingcited by 4
  9. 2025Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Modelscited by 3
  10. 2025A Linguistics-Aware LLM Watermarking via Syntactic Predictabilitycited by 2
  11. 2025Building from Scratch: A Multi-Agent Framework with Human-in-the-Loop for Multilingual Legal Terminology Mappingcited by 2
  12. 2026Who Checks the Citations? Benchmarking Legal Hallucination Detectioncited by 2
16 more
  1. 2025Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledgecited by 1
  2. 2025CrunchLLM: Multitask LLMs for Structured Business Reasoning and Outcome Predictioncited by 1
  3. 2026Structured Decomposition for LLM Reasoning: Cross-Domain Validation and Semantic Web Integrationcited by 1
  4. 2026Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Lawcited by 1
  5. 2026AdversaRiskQA: An Adversarial Factuality Benchmark for High-Risk Domainscited by 1
  6. 2026CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrievalcited by 1
  7. 2026Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedbackcited by 1
  8. 2026KnowHal: A Knowledge-Driven Benchmark for Comprehensive Multimodal Hallucination Evaluationcited by 1
  9. 2026Legal LLM Hallucination Should Be Evaluated as Failure of Legal Warrantcited by 1
  10. 2026From Atoms to Chains: Divergence-Guided Reasoning Curriculum for Unlabeled LLM Domain Adaptationcited by 0
  11. 2026Supervised Fine-Tuning Needs to Unlock the Potential of Token Prioritycited by 0
  12. 2026Enhancing Legal LLMs through Metadata-Enriched RAG Pipelines and Direct Preference Optimizationcited by 0
  13. 2026PoliLegalLM: A Technical Report on a Large Language Model for Political and Legal Affairscited by 0
  14. 2026How Much Do Legal RAG Systems Still Hallucinate?cited by 0
  15. 2026InsufficiencyBench: Evaluating LLM legal advice on underspecified user queriescited by 0
  16. 2026LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation
  2. 2023DelucionQA: Detecting Hallucinations in Domain-specific Question Answering
  3. 2025Beyond Facts: Evaluating Intent Hallucination in Large Language Models
  4. 2026When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection
  5. 2025QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting
  6. 2023ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks
  7. 2025Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilization
  8. 2024Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem