# A Pragmatic Guide to Geoparsing Evaluation

arXiv: https://arxiv.org/abs/1810.12368 · HTML: https://mlsanity.com/paper/1810.12368/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2018.

## Built on by (most cited first, 17 of 17)

- [GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding](https://mlsanity.com/paper/2310.14478/) (2023)
- [Incidents1M: a large-scale dataset of images with natural disasters, damage, and incidents](https://mlsanity.com/paper/2201.04236/) (2022)
- [Dataset Geography: Mapping Language Data to Language Users](https://mlsanity.com/paper/2112.03497/) (2021)
- [Location reference recognition from texts: A survey and comparison](https://mlsanity.com/paper/2207.01683/) (2022)
- [GeoReasoner: Reasoning On Geospatially Grounded Context For Natural Language Understanding](https://mlsanity.com/paper/2408.11366/) (2024)
- [Improving Toponym Resolution with Better Candidate Generation, Transformer-based Reranking, and Two-Stage Resolution](https://mlsanity.com/paper/2305.11315/) (2023)
- [Mordecai 3: A Neural Geoparser and Event Geocoder](https://mlsanity.com/paper/2303.13675/) (2023)
- [Leave no Place Behind: Improved Geolocation in Humanitarian Documents](https://mlsanity.com/paper/2309.02914/) (2023)
- [Arukikata Travelogue Dataset with Geographic Entity Mention, Coreference, and Link Annotation](https://mlsanity.com/paper/2305.13844/) (2023)
- [DIGMAPPER: A Modular System for Automated Geologic Map Digitization](https://mlsanity.com/paper/2506.16006/) (2025)
- [Geolocating News about Extreme Climate Events: A Comparative Analysis of Off-the-Shelf Tools for Toponym Identification in German](https://mlsanity.com/paper/2605.03414/) (2026)
- [Graph-Structured Trajectory Extraction from Travelogues](https://mlsanity.com/paper/2410.16633/) (2024)
- [Large Language Models for Geolocation Extraction in Humanitarian Crisis Response](https://mlsanity.com/paper/2602.08872/) (2026)
- [Assessing socio-economic climate impacts from text data](https://mlsanity.com/paper/2605.20793/) (2026)
- [RACCOON: A Retrieval-Augmented Generation Approach for Location Coordinate Capture from News Articles](https://mlsanity.com/paper/2501.11440/) (2025)
- [ATD-Trans: A Geographically Grounded Japanese-English Travelogue Translation Dataset](https://mlsanity.com/paper/2605.12933/) (2026)
- [Analyzing Public Discourse on Urbanism: Topic Clustering, Sentiment Analysis and Retrieval-Augmented Generation using YouTube Comments](https://mlsanity.com/paper/2609.22705/) (2026)

## Abstract (arXiv)

Empirical methods in geoparsing have thus far lacked a standard evaluation framework describing the task, metrics and data used to compare state-of-the-art systems. Evaluation is further made inconsistent, even unrepresentative of real-world usage by the lack of distinction between the different types of toponyms, which necessitates new guidelines, a consolidation of metrics and a detailed toponym taxonomy with implications for Named Entity Recognition (NER) and beyond. To address these deficiencies, our manuscript introduces a new framework in three parts. Part 1) Task Definition: clarified via corpus linguistic analysis proposing a fine-grained Pragmatic Taxonomy of Toponyms. Part 2) Metrics: discussed and reviewed for a rigorous evaluation including recommendations for NER/Geoparsing practitioners. Part 3) Evaluation Data: shared via a new dataset called GeoWebNews to provide test/train examples and enable immediate use of our contributions. In addition to fine-grained Geotagging and Toponym Resolution (Geocoding), this dataset is also suitable for prototyping and evaluating machine learning NLP models.