# LocalBench: Benchmarking LLMs on County-Level Local Knowledge and Reasoning

arXiv: https://arxiv.org/abs/2511.10459 · HTML: https://mlsanity.com/paper/2511.10459/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 10 of 10)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPT](https://mlsanity.com/paper/2307.07930/) (2023)
- [TravelAgent: An AI Assistant for Personalized Travel Planning](https://mlsanity.com/paper/2409.08069/) (2024)
- [Quite Good, but Not Enough: Nationality Bias in Large Language Models -- A Case Study of ChatGPT](https://mlsanity.com/paper/2405.06996/) (2024)
- [TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Time](https://mlsanity.com/paper/2501.07482/) (2025)
- [A Turing Test for ''Localness'': Conceptualizing, Defining, and Recognizing Localness in People and Machines](https://mlsanity.com/paper/2505.07282/) (2025)
- [Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplication](https://mlsanity.com/paper/2506.22714/) (2025)

## Built on by (most cited first, 3 of 3)

- [Curi'o-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining](https://mlsanity.com/paper/2512.12770/) (2025)
- [Collective Narrative Grounding: Community-Coordinated Data Contributions to Improve Local AI Systems](https://mlsanity.com/paper/2601.04201/) (2025)
- [CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information](https://mlsanity.com/paper/2609.08094/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been widely evaluated on macro-scale geographic tasks, such as global factual recall, event summarization, and regional reasoning. Yet, their ability to handle hyper-local knowledge remains poorly understood. This gap is increasingly consequential as real-world applications, from civic platforms to community journalism, demand AI systems that can reason about neighborhood-specific dynamics, cultural narratives, and local governance. Existing benchmarks fall short in capturing this complexity, often relying on coarse-grained data or isolated references. We present LocalBench, the first benchmark designed to systematically evaluate LLMs on county-level local knowledge across the United States. Grounded in the Localness Conceptual Framework, LocalBench includes 14,782 validated question-answer pairs across 526 U.S. counties in 49 states, integrating diverse sources such as Census statistics, local subreddit discourse, and regional news. It spans physical, cognitive, and relational dimensions of locality. Using LocalBench, we evaluate 13 state-of-the-art LLMs under both closed-book and web-augmented settings. Our findings reveal critical limitations: even the best-performing models reach only 56.8% accuracy on narrative-style questions and perform below 15.5% on numerical reasoning. Moreover, larger model size and web augmentation do not guarantee better performance, for example, search improves Gemini's accuracy by +13.6%, but reduces GPT-series performance by -11.4%. These results underscore the urgent need for language models that can support equitable, place-aware AI systems: capable of engaging with the diverse, fine-grained realities of local communities across geographic and cultural contexts.