# Unpacking Large Language Models with Conceptual Consistency

arXiv: https://arxiv.org/abs/2209.15093 · HTML: https://mlsanity.com/paper/2209.15093/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 29 Sep 2022.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)
- [Explaining Classifiers with Causal Concept Effect (CaCE)](https://mlsanity.com/paper/1907.07165/) (2019)
- [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022)
- [On the Paradox of Learning to Reason from Data](https://mlsanity.com/paper/2205.11502/) (2022)
- [Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs](https://mlsanity.com/paper/2111.13654/) (2021)
- [Meaning without reference in large language models](https://mlsanity.com/paper/2208.02957/) (2022)
- [Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering](https://mlsanity.com/paper/1910.14087/) (2019)
- [Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation](https://mlsanity.com/paper/1909.04696/) (2019)
- [Human-Centered Concept Explanations for Neural Networks](https://mlsanity.com/paper/2202.12451/) (2022)
- [Accurate, yet inconsistent? Consistency Analysis on Language Understanding Models](https://mlsanity.com/paper/2108.06665/) (2021)
- [It Takes Two to Tango: Towards Theory of AI's Mind](https://mlsanity.com/paper/1704.00717/) (2017)

## Built on by (most cited first, 18 of 18)

- [Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate](https://mlsanity.com/paper/2305.11595/) (2023)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Moral Foundations of Large Language Models](https://mlsanity.com/paper/2310.15337/) (2023)
- [Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models](https://mlsanity.com/paper/2309.04461/) (2023)
- [Reasoning Runtime Behavior of a Program with LLM: How Far Are We?](https://mlsanity.com/paper/2403.16437/) (2024)
- [Probing Conceptual Understanding of Large Visual-Language Models](https://mlsanity.com/paper/2304.03659/) (2023)
- [If Concept Bottlenecks are the Question, are Foundation Models the Answer?](https://mlsanity.com/paper/2504.19774/) (2025)
- [Do language models have coherent mental models of everyday things?](https://mlsanity.com/paper/2212.10029/) (2022)
- [Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflection](https://mlsanity.com/paper/2501.15355/) (2025)
- [Improving Consistency in Large Language Models through Chain of Guidance](https://mlsanity.com/paper/2502.15924/) (2025)
- [Can Large Language Models Help Experimental Design for Causal Discovery?](https://mlsanity.com/paper/2503.01139/) (2025)
- [OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Ontological Understanding, Reasoning and Learning](https://mlsanity.com/paper/2505.11031/) (2025)
- [BloomVQA: Assessing Hierarchical Multi-modal Comprehension](https://mlsanity.com/paper/2312.12716/) (2023)
- [Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responses](https://mlsanity.com/paper/2412.15683/) (2024)
- [Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations](https://mlsanity.com/paper/2605.16405/) (2026)
- [Concept-Oriented Deep Learning with Large Language Models](https://mlsanity.com/paper/2306.17089/) (2023)
- [On the Brittleness of LLMs: A Journey around Set Membership](https://mlsanity.com/paper/2511.12728/) (2025)
- [How Do Large Language Models Learn Concepts During Continual Pre-Training?](https://mlsanity.com/paper/2601.03570/) (2026)

## Abstract (arXiv)

If a Large Language Model (LLM) answers "yes" to the question "Are mountains tall?" then does it know what a mountain is? Can you rely on it responding correctly or incorrectly to other questions about mountains? The success of Large Language Models (LLMs) indicates they are increasingly able to answer queries like these accurately, but that ability does not necessarily imply a general understanding of concepts relevant to the anchor query. We propose conceptual consistency to measure a LLM's understanding of relevant concepts. This novel metric measures how well a model can be characterized by finding out how consistent its responses to queries about conceptually relevant background knowledge are. To compute it we extract background knowledge by traversing paths between concepts in a knowledge base and then try to predict the model's response to the anchor query from the background knowledge. We investigate the performance of current LLMs in a commonsense reasoning setting using the CSQA dataset and the ConceptNet knowledge base. While conceptual consistency, like other metrics, does increase with the scale of the LLM used, we find that popular models do not necessarily have high conceptual consistency. Our analysis also shows significant variation in conceptual consistency across different kinds of relations, concepts, and prompts. This serves as a step toward building models that humans can apply a theory of mind to, and thus interact with intuitively.