# Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding

arXiv: https://arxiv.org/abs/2608.07353 · HTML: https://mlsanity.com/paper/2608.07353/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks](https://mlsanity.com/paper/1801.03454/) (2018)
- [Does CLIP Bind Concepts? Probing Compositionality in Large Image Models](https://mlsanity.com/paper/2212.10537/) (2022)
- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization](https://mlsanity.com/paper/2306.05064/) (2023)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [ICLR: In-Context Learning of Representations](https://mlsanity.com/paper/2501.00070/) (2024)
- [Linear Spaces of Meanings: Compositional Structures in Vision-Language Models](https://mlsanity.com/paper/2302.14383/) (2023)
- [From task structures to world models: What do LLMs know?](https://mlsanity.com/paper/2310.04276/) (2023)
- [Geographic Question Answering: Challenges, Uniqueness, Classification, and Future Directions](https://mlsanity.com/paper/2105.09392/) (2021)
- [Unit Testing for Concepts in Neural Networks](https://mlsanity.com/paper/2208.10244/) (2022)
- [Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations](https://mlsanity.com/paper/2505.17136/) (2025)
- [MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models](https://mlsanity.com/paper/2501.00316/) (2024)
- [Can Language Models Understand Physical Concepts?](https://mlsanity.com/paper/2305.14057/) (2023)
- [Leveraging LLMs for Unsupervised Dense Retriever Ranking](https://mlsanity.com/paper/2402.04853/) (2024)
- [Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing](https://mlsanity.com/paper/2501.14457/) (2025)
- [GeoAnalystBench: A GeoAI benchmark for assessing large language models for spatial analysis workflow and code generation](https://mlsanity.com/paper/2509.05881/) (2025)
- [Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare](https://mlsanity.com/paper/2502.13319/) (2025)
- [Evaluating Brain-Inspired Modular Training in Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2401.03646/) (2024)
- [Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddings](https://mlsanity.com/paper/2105.08585/) (2021)
- [Counterfactual Collaborative Reasoning](https://mlsanity.com/paper/2307.00165/) (2023)
- [MechIR: A Mechanistic Interpretability Framework for Information Retrieval](https://mlsanity.com/paper/2501.10165/) (2025)
- [Evaluating List Construction and Temporal Understanding capabilities of Large Language Models](https://mlsanity.com/paper/2506.21783/) (2025)

## Abstract (arXiv)

Understanding concepts is fundamental to generalization. Despite their impressive performance on a wide range of tasks, Large Language Models (LLMs) still struggle with genuine concept understanding. Prior work has evaluated conceptual understanding in LLMs using natural-language benchmarks or narrowly scoped synthetic tasks, but these settings often conflate multiple skills or lack precise control over the underlying concepts and their properties. To support controlled probing of concepts in LLMs, we design tests on their core properties: abstraction, compositionality, and groundness. We set up a concept-centric benchmark, targeting spatial concepts such as direction, distance, topology, and their compositions, and use question answering tasks serving as a proxy. We conduct extensive experiments across multiple LLM architectures and training regimes to analyze how model scale and design impact conceptual understanding. The results reveal clear limitations in current LLMs and provide insights into the factors shaping their ability to acquire and compose structured concepts. Our findings shed light on how concept-based LLMs can be redesigned for improved information access and knowledge management. The code will be available at https://github.com/rd20karim/concept-probing.