# Toward best research practices in AI Psychology

arXiv: https://arxiv.org/abs/2312.01276 · HTML: https://mlsanity.com/paper/2312.01276/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 14 references that are in the map.
- First posted on arXiv 3 Dec 2023.

## Builds on (most cited first, 3 of 3)

- [Can language models handle recursively nested grammatical structures? A case study on comparing models and humans](https://mlsanity.com/paper/2210.15303/) (2022)
- [Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema](https://mlsanity.com/paper/2104.08161/) (2021)
- [How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG](https://mlsanity.com/paper/1811.01778/) (2018)

## Built on by (most cited first, 8 of 8)

- [Evaluating the Robustness of Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2411.14215/) (2024)
- [How Can LLM Guide RL? A Value-Based Approach](https://mlsanity.com/paper/2402.16181/) (2024)
- [Incremental Comprehension of Garden-Path Sentences by Large Language Models: Semantic Interpretation, Syntactic Re-Analysis, and Attention](https://mlsanity.com/paper/2405.16042/) (2024)
- [What is it for a Machine Learning Model to Have a Capability?](https://mlsanity.com/paper/2405.08989/) (2024)
- [Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers](https://mlsanity.com/paper/2410.11672/) (2024)
- [Do Large Language Models Solve ARC Visual Analogies Like People Do?](https://mlsanity.com/paper/2403.09734/) (2024)
- [On the Context Sensitivity of LLM Moral Judgment](https://mlsanity.com/paper/2603.23114/) (2026)
- [What Makes an Evaluation Useful? Common Pitfalls and Best Practices](https://mlsanity.com/paper/2503.23424/) (2025)

## Abstract (arXiv)

Language models have become an essential part of the burgeoning field of AI Psychology. I discuss 14 methodological considerations that can help design more robust, generalizable studies evaluating the cognitive abilities of language-based AI systems, as well as to accurately interpret the results of these studies.