The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost
On the map: LLM reasoning → Mathematical reasoning
Abstract (arXiv)
Strategic model selection and reasoning settings are more effective than ensembling for optimizing automated scoring with large language models (LLMs). We examined self-consistency (intra-model majority voting) and reasoning effort for scoring conversation-based assessment items in high school mathematics, evaluating 900 student conversations against human-scored ground truths using frontier and low-cost models from OpenAI and Google. Temperature sampling significantly improved accuracy over deterministic calls, but increasing ensemble size (j = 1 to 7) produced no significant gains. Higher reasoning effort showed a significant positive linear trend with scoring accuracy, though the benefit varied by model family. An efficiency frontier analysis identified Gemini 3.1 Pro Preview at low reasoning as the most accurate but costly configuration; GPT-5.4 Nano and Mini with no reasoning offered the best cost-performance balance.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 5 of its 12 references that are in the map. First posted on arXiv 3 Apr 2026.
Builds on
5 of 12 references.
References · 5
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
- 2023Applying Large Language Models and Chain-of-Thought for Automatic Scoringcited by 68
- 2024Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability to Mark Short Answer Questions in K-12 Educationcited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
- 2025MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoning
- 2026Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks
- 2024Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring
- 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
- 2024LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs
- 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
- 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitions