Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Predicting human item difficulty is central to educational assessment, where reliable estimates support fairness and effective test construction. Existing methods often depend on costly human calibration or item-level textual representations, providing limited evidence about the cognitive processes that make items difficult. We argue that difficulty should be viewed not only as a property of item text, but also as an observable consequence of the problem-solving burden an item induces. Large Reasoning Models (LRMs) offer scalable process evidence through reasoning traces, but such evidence must be structured to support interpretable modeling. To this end, we introduce Epi2Diff (Episode to Difficulty), a framework that maps LRM reasoning traces into cognitively grounded episode sequences. These episodes group trace segments into functional problem-solving states, enabling difficulty to be modeled through reasoning scale, effort allocation, and state transitions. Epi2Diff extracts compact episode-dynamic features and combines them with semantic item representations for human difficulty prediction. Experiments on four real-world human difficulty datasets show that Epi2Diff consistently outperforms strong baselines, including fine-tuned small language models, LLM in-context learning, and supervised LLM adaptation. On SAT-derived classification benchmarks, Epi2Diff achieves an 8.1% average relative gain over supervised LLM fine-tuning baselines. Further analyses show that harder items induce more effortful, iterative, and implementation-centered episode dynamics, rather than merely longer responses. These results demonstrate that cognitive episodes in LRM reasoning traces provide a predictive and interpretable process representation for human item difficulty, offering a new lens for educational measurement with reasoning models.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 28 of its 70 references that are in the map. First posted on arXiv 26 Jun 2026.
Builds on
28 of 70 references.
Closest ancestors · 4
Same direction, newest first
- 2025Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?9 Apr
- 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning6 Mar
- 2025Self-rewarding correction for mathematical reasoning26 Feb
- 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
Other references · 24
- 2023GPT-4 Technical Reportcited by 18,132
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2025Qwen3 Technical Reportcited by 8,195
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
- 2024OpenAI o1 System Cardcited by 1,220
- 2025OpenAI GPT-5 System Cardcited by 910
- 2024Phi-4 Technical Reportcited by 675
- 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
- 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
12 more
- 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
- 2025Reasoning Language Models: A Blueprintcited by 41
- 2020ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generatorscited by 26
- 2025What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoTcited by 24
- 2025Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Reviewcited by 10
- 2025Schoenfeld's Anatomy of Mathematical Reasoning by Language Modelscited by 8
- 2024Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimationcited by 7
- 2025Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Predictioncited by 7
- 2026Towards Valid Student Simulation with Large Language Modelscited by 7
- 2025Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithmscited by 6
- 2025LongReasonArena: A Long Reasoning Benchmark for Large Language Modelscited by 3
- 2026LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessmentcited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Humans Disengage, Reasoning Models Persist: Separating Difficulty Registration from Deliberation Allocation
- 2025Cognitive Foundations for Reasoning and Their Manifestation in LLMs
- 2026Behavior-Aware Item Modeling via Dynamic Procedural Solution Representations for Knowledge Tracing
- 2025Understanding the Thinking Process of Reasoning Models: A Perspective from Schoenfeld's Episode Theory
- 2025Reasoning Steps as Curriculum: Using Depth of Thought as a Difficulty Signal for Tuning LLMs
- 2026KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty
- 2025Probing the "Psyche'' of Large Reasoning Models: Understanding Through a Human Lens
- 2025LLMs Encode How Difficult Problems Are