# Objectifying the Subjective: Cognitive Biases in Topic Interpretations

arXiv: https://arxiv.org/abs/2507.19117 · HTML: https://mlsanity.com/paper/2507.19117/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 110 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Builds on (most cited first, 16 of 16)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI](https://mlsanity.com/paper/2204.01075/) (2022)
- [NLPositionality: Characterizing Design Biases of Datasets and Models](https://mlsanity.com/paper/2306.01943/) (2023)
- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [POTATO: The Portable Text Annotation Tool](https://mlsanity.com/paper/2212.08620/) (2022)
- [How to Make Causal Inferences Using Texts](https://mlsanity.com/paper/1802.02163/) (2018)
- [User Simulation for Evaluating Information Access Systems](https://mlsanity.com/paper/2306.08550/) (2023)
- [The Authenticity Gap in Human Evaluation](https://mlsanity.com/paper/2205.11930/) (2022)
- [Text-Based Ideal Points](https://mlsanity.com/paper/2005.04232/) (2020)
- [Contextualized Topic Coherence Metrics](https://mlsanity.com/paper/2305.14587/) (2023)
- [Towards a Quantum-Like Cognitive Architecture for Decision-Making](https://mlsanity.com/paper/1905.05176/) (2019)
- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [Are Neural Topic Models Broken?](https://mlsanity.com/paper/2210.16162/) (2022)
- [SOTorrent: Studying the Origin, Evolution, and Usage of Stack Overflow Code Snippets](https://mlsanity.com/paper/1809.02814/) (2018)
- [Cross-Dataset Design Discussion Mining](https://mlsanity.com/paper/2001.01424/) (2020)

## Abstract (arXiv)

Interpretation of topics is crucial for their downstream applications. State-of-the-art evaluation measures of topic quality such as coherence and word intrusion do not measure how much a topic facilitates the exploration of a corpus. To design evaluation measures grounded on a task, and a population of users, we do user studies to understand how users interpret topics. We propose constructs of topic quality and ask users to assess them in the context of a topic and provide rationale behind evaluations. We use reflexive thematic analysis to identify themes of topic interpretations from rationales. Users interpret topics based on availability and representativeness heuristics rather than probability. We propose a theory of topic interpretation based on the anchoring-and-adjustment heuristic: users anchor on salient words and make semantic adjustments to arrive at an interpretation. Topic interpretation can be viewed as making a judgment under uncertainty by an ecologically rational user, and hence cognitive biases aware user models and evaluation frameworks are needed.