# Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice

arXiv: https://arxiv.org/abs/2409.07713 · HTML: https://mlsanity.com/paper/2409.07713/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [Query Rewriting for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2305.14283/) (2023)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://mlsanity.com/paper/2310.01352/) (2023)
- [How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence](https://mlsanity.com/paper/2004.12158/) (2020)
- [Bridging the Preference Gap between Retrievers and LLMs](https://mlsanity.com/paper/2401.06954/) (2024)
- [Recitation-Augmented Language Models](https://mlsanity.com/paper/2210.01296/) (2022)
- [Legal Prompting: Teaching a Language Model to Think Like a Lawyer](https://mlsanity.com/paper/2212.01326/) (2022)
- [Understanding Retrieval Augmentation for Long-Form Question Answering](https://mlsanity.com/paper/2310.12150/) (2023)
- [Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction](https://mlsanity.com/paper/2307.08321/) (2023)
- [Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts](https://mlsanity.com/paper/2305.04417/) (2023)
- [Building Efficient Universal Classifiers with Natural Language Inference](https://mlsanity.com/paper/2312.17543/) (2023)
- [The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate](https://mlsanity.com/paper/2402.06204/) (2024)
- [Parameter-Efficient Legal Domain Adaptation](https://mlsanity.com/paper/2210.13712/) (2022)

## Built on by (most cited first, 5 of 5)

- [Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models](https://mlsanity.com/paper/2508.12461/) (2025)
- [RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelines](https://mlsanity.com/paper/2505.13538/) (2025)
- [How Accurate Are LLMs at Multi-Question Answering on Conversational Transcripts?](https://mlsanity.com/paper/2509.21732/) (2025)
- [CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law](https://mlsanity.com/paper/2605.30497/) (2026)
- [Gender Bias in LLMs: Preliminary Evidence from Shared Parenting Scenario in Czech Family Law](https://mlsanity.com/paper/2601.05879/) (2026)

## Abstract (arXiv)

Generative AI models, such as the GPT and Llama series, have significant potential to assist laypeople in answering legal questions. However, little prior work focuses on the data sourcing, inference, and evaluation of these models in the context of laypersons. To this end, we propose a human-centric legal NLP pipeline, covering data sourcing, inference, and evaluation. We introduce and release a dataset, LegalQA, with real and specific legal questions spanning from employment law to criminal law, corresponding answers written by legal experts, and citations for each answer. We develop an automatic evaluation protocol for this dataset, then show that retrieval-augmented generation from only 850 citations in the train set can match or outperform internet-wide retrieval, despite containing 9 orders of magnitude less data. Finally, we propose future directions for open-sourced efforts, which fall behind closed-sourced models.