# Cognitive network science reveals bias in GPT-3, ChatGPT, and GPT-4 mirroring math anxiety in high-school students

arXiv: https://arxiv.org/abs/2305.18320 · HTML: https://mlsanity.com/paper/2305.18320/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 67 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [Crowdsourcing a Word-Emotion Association Lexicon](https://mlsanity.com/paper/1308.6297/) (2013)
- [Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models](https://mlsanity.com/paper/2304.03738/) (2023)
- [Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings](https://mlsanity.com/paper/1904.04047/) (2019)
- [The Debate Over Understanding in AI's Large Language Models](https://mlsanity.com/paper/2210.13966/) (2022)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Understanding Human Intelligence through Human Limitations](https://mlsanity.com/paper/2009.14050/) (2020)
- [Intersectional Bias in Causal Language Models](https://mlsanity.com/paper/2107.07691/) (2021)
- [Towards an Enhanced Understanding of Bias in Pre-trained Neural Language Models: A Survey with Special Emphasis on Affective Bias](https://mlsanity.com/paper/2204.10365/) (2022)
- [Why we need biased AI -- How including cognitive and ethical machine biases can enhance AI systems](https://mlsanity.com/paper/2203.09911/) (2022)
- [Mapping computational thinking mindsets between educational levels with cognitive network science](https://mlsanity.com/paper/2007.09402/) (2020)
- [Feature-rich multiplex lexical networks reveal mental strategies of early language learning](https://mlsanity.com/paper/2201.05061/) (2022)
- [Text characterization based on recurrence networks](https://mlsanity.com/paper/2201.06665/) (2022)

## Built on by (most cited first, 2 of 2)

- [CBEval: A framework for evaluating and interpreting cognitive biases in LLMs](https://mlsanity.com/paper/2412.03605/) (2024)
- [ChatGPT & Mechanical Engineering: Examining performance on the FE Mechanical Engineering and Undergraduate Exams](https://mlsanity.com/paper/2309.15866/) (2023)

## Abstract (arXiv)

Large language models are becoming increasingly integrated into our lives. Hence, it is important to understand the biases present in their outputs in order to avoid perpetuating harmful stereotypes, which originate in our own flawed ways of thinking. This challenge requires developing new benchmarks and methods for quantifying affective and semantic bias, keeping in mind that LLMs act as psycho-social mirrors that reflect the views and tendencies that are prevalent in society. One such tendency that has harmful negative effects is the global phenomenon of anxiety toward math and STEM subjects. Here, we investigate perceptions of math and STEM fields provided by cutting-edge language models, namely GPT-3, Chat-GPT, and GPT-4, by applying an approach from network science and cognitive psychology. Specifically, we use behavioral forma mentis networks (BFMNs) to understand how these LLMs frame math and STEM disciplines in relation to other concepts. We use data obtained by probing the three LLMs in a language generation task that has previously been applied to humans. Our findings indicate that LLMs have an overall negative perception of math and STEM fields, with math being perceived most negatively. We observe significant differences across the three LLMs. We observe that newer versions (i.e. GPT-4) produce richer, more complex perceptions as well as less negative perceptions compared to older versions and N=159 high-school students. These findings suggest that advances in the architecture of LLMs may lead to increasingly less biased models that could even perhaps someday aid in reducing harmful stereotypes in society rather than perpetuating them.