# Right Number, Wrong State? Measuring Cross-Jurisdiction Substitution in LLM Recall of State Policy

arXiv: https://arxiv.org/abs/2610.09458 · HTML: https://mlsanity.com/paper/2610.09458/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 22 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 5 of 5)

- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [The Law of Knowledge Overshadowing: Towards Understanding, Predicting, and Preventing LLM Hallucination](https://mlsanity.com/paper/2502.16143/) (2025)
- [Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries](https://mlsanity.com/paper/2511.06700/) (2025)
- [CrossLex: A Source-Grounded Benchmark for Cross-Jurisdictional Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2608.01292/) (2026)
- [Legal LLM Hallucination Should Be Evaluated as Failure of Legal Warrant](https://mlsanity.com/paper/2609.17546/) (2026)

## Abstract (arXiv)

When an LLM answers a state-specific policy question wrongly, it may be hallucinating, or it may be returning a real value that holds in another state. We test this with a minimal-set design: the question wording is fixed and only the jurisdiction varies, across the 50 U.S. states and the District of Columbia (51 jurisdictions) and three exactly defined Medicaid income-eligibility quantities. Gold values come from an official data book and agree with an independent source in 101 of 102 checked cells. Under a pre-registered protocol, Claude Sonnet 5.5 and GPT-5.6 Sol reproducibly give another state's current value, identical across two independent repeats, for 10 and 25 of 153 items. The count depends on the reference set: crediting any wrong answer that equals another state's value yields 3-5x more reproducible substitutions than checking every number in the asked state's own records, because many apparent cross-state answers are the asked state's own values under another convention or from an earlier year. Claims about cross-jurisdiction error need a complete same-state reference set. We will release the protocol, gold table, and all model outputs.