# Scaling up the think-aloud method

arXiv: https://arxiv.org/abs/2505.23931 · HTML: https://mlsanity.com/paper/2505.23931/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 5 of its 33 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 5 of 5)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Stream of Search (SoS): Learning to Search in Language](https://mlsanity.com/paper/2404.03683/) (2024)

## Built on by (most cited first, 8 of 8)

- [Learning User Simulators with Turing Rewards](https://mlsanity.com/paper/2606.19336/) (2026)
- [Code-enabled language models can outperform reasoning models on diverse tasks](https://mlsanity.com/paper/2510.20909/) (2025)
- [HugAgent: A Human Simulation Benchmark for Individual-Level Reasoning](https://mlsanity.com/paper/2510.15144/) (2025)
- [Cognitive Models and AI Algorithms Provide Templates for Designing Language Agents](https://mlsanity.com/paper/2602.22523/) (2026)
- [Think-Aloud Reshapes Automated Cognitive Model Discovery Beyond Behavior](https://mlsanity.com/paper/2605.05091/) (2026)
- [Leveraging Speech to Identify Signatures of Insight and Transfer in Problem Solving](https://mlsanity.com/paper/2605.12970/) (2026)
- [Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?](https://mlsanity.com/paper/2603.27694/) (2026)
- [Rewarding Efficient Reasoning Improves Abstention on Underspecified Tasks in Reasoning Models](https://mlsanity.com/paper/2609.20846/) (2026)

## Abstract (arXiv)

The think-aloud method, where participants voice their thoughts as they solve a task, is a valuable source of rich data about human reasoning processes. Yet, it has declined in popularity in contemporary cognitive science, largely because labor-intensive transcription and annotation preclude large sample sizes. Here, we develop methods to automate the transcription and annotation of verbal reports of reasoning using natural language processing tools, allowing for large-scale analysis of think-aloud data. In our study, 640 participants thought aloud while playing the Game of 24, a mathematical reasoning task. We automatically transcribed the recordings and coded the transcripts as search graphs, finding moderate inter-rater reliability with humans. We analyze these graphs and characterize consistency and variation in human reasoning traces. Our work demonstrates the value of think-aloud data at scale and serves as a proof of concept for the automated analysis of verbal reports.