# Do Language Models Know the Way to Rome?

arXiv: https://arxiv.org/abs/2109.07971 · HTML: https://mlsanity.com/paper/2109.07971/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://mlsanity.com/paper/1805.03620/) (2018)
- [Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains](https://mlsanity.com/paper/1906.01539/) (2019)
- [Are All Good Word Vector Spaces Isomorphic?](https://mlsanity.com/paper/2004.04070/) (2020)
- [Are distributional representations ready for the real world? Evaluating word vectors for grounded perceptual meaning](https://mlsanity.com/paper/1705.11168/) (2017)
- [Intrinsic Subspace Evaluation of Word Embedding Representations](https://mlsanity.com/paper/1606.07902/) (2016)

## Built on by (most cited first, 18 of 18)

- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Eight Things to Know about Large Language Models](https://mlsanity.com/paper/2304.00612/) (2023)
- [Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models](https://mlsanity.com/paper/2310.16570/) (2023)
- [Are Large Language Models Geospatially Knowledgeable?](https://mlsanity.com/paper/2310.13002/) (2023)
- [What Do Language Models Hear? Probing for Auditory Representations in Language Models](https://mlsanity.com/paper/2402.16998/) (2024)
- [Monotonic Representation of Numeric Properties in Language Models](https://mlsanity.com/paper/2403.10381/) (2024)
- [Spatial-RAG: Spatial Retrieval Augmented Generation for Real-World Geospatial Reasoning Questions](https://mlsanity.com/paper/2502.18470/) (2025)
- [Around the World in 24 Hours: Probing LLM Knowledge of Time and Place](https://mlsanity.com/paper/2506.03984/) (2025)
- [Linear Spatial World Models Emerge in Large Language Models](https://mlsanity.com/paper/2506.02996/) (2025)
- [On Entity Identification in Language Models](https://mlsanity.com/paper/2506.02701/) (2025)
- [Geospatial Mechanistic Interpretability of Large Language Models](https://mlsanity.com/paper/2505.03368/) (2025)
- [Where meaning lives: Layer-wise accessibility of psycholinguistic features in encoder and decoder language models](https://mlsanity.com/paper/2601.03798/) (2026)
- [Ered: Enhanced Text Representations with Entities and Descriptions](https://mlsanity.com/paper/2208.08954/) (2022)
- [Can LLMs Learn to Map the World from Local Descriptions?](https://mlsanity.com/paper/2505.20874/) (2025)
- [Towards Applying Large Language Models to Complement Single-Cell Foundation Models](https://mlsanity.com/paper/2507.10039/) (2025)
- [Explaining Black-box Language Models with Knowledge Probing Systems: A Post-hoc Explanation Perspective](https://mlsanity.com/paper/2508.16969/) (2025)
- [Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Models](https://mlsanity.com/paper/2511.04053/) (2025)
- [Recovering Temporal and Geographic Signals from Language Model Embeddings](https://mlsanity.com/paper/2609.05721/) (2026)

## Abstract (arXiv)

The global geometry of language models is important for a range of applications, but language model probes tend to evaluate rather local relations, for which ground truths are easily obtained. In this paper we exploit the fact that in geography, ground truths are available beyond local relations. In a series of experiments, we evaluate the extent to which language model representations of city and country names are isomorphic to real-world geography, e.g., if you tell a language model where Paris and Berlin are, does it know the way to Rome? We find that language models generally encode limited geographic information, but with larger models performing the best, suggesting that geographic knowledge can be induced from higher-order co-occurrence statistics.