# Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries

arXiv: https://arxiv.org/abs/2511.06700 · HTML: https://mlsanity.com/paper/2511.06700/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Builds on (most cited first, 12 of 12)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [Integrative Decoding: Improve Factuality via Implicit Self-consistency](https://mlsanity.com/paper/2410.01556/) (2024)
- [Fine-Grained Self-Endorsement Improves Factuality and Reasoning](https://mlsanity.com/paper/2402.15631/) (2024)
- [The Pitfalls of Defining Hallucination](https://mlsanity.com/paper/2401.07897/) (2024)
- [LLMs & Legal Aid: Understanding Legal Needs Exhibited Through User Queries](https://mlsanity.com/paper/2501.01711/) (2025)

## Built on by (most cited first, 3 of 3)

- [Legal LLM Hallucination Should Be Evaluated as Failure of Legal Warrant](https://mlsanity.com/paper/2609.17546/) (2026)
- [Assessing and Explaining the Persuadability of Large Language Models as Legal Decision Tools](https://mlsanity.com/paper/2604.26233/) (2026)
- [Right Number, Wrong State? Measuring Cross-Jurisdiction Substitution in LLM Recall of State Policy](https://mlsanity.com/paper/2610.09458/) (2026)

## Abstract (arXiv)

How do we make a meaningful comparison of a large language model's knowledge of the law in one place compared to another? Quantifying these differences is critical to understanding if the quality of the legal information obtained by users of LLM-based chatbots varies depending on their location. However, obtaining meaningful comparative metrics is challenging because legal institutions in different places are not themselves easily comparable. In this work we propose a methodology to obtain place-to-place metrics based on the comparative law concept of functionalism. We construct a dataset of factual scenarios drawn from Reddit posts by users seeking legal advice for family, housing, employment, crime and traffic issues. We use these to elicit a summary of a law from the LLM relevant to each scenario in Los Angeles, London and Sydney. These summaries, typically of a legislative provision, are manually evaluated for hallucinations. We show that the rate of hallucination of legal information by leading closed-source LLMs is significantly associated with place. This suggests that the quality of legal solutions provided by these models is not evenly distributed across geography. Additionally, we show a strong negative correlation between hallucination rate and the frequency of the majority response when the LLM is sampled multiple times, suggesting a measure of uncertainty of model predictions of legal facts.