Finding Answers in Thought Matters: Revisiting Evaluation on Large Language Models with Reasoning
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Evaluating generative models, such as large language models (LLMs), commonly involves question-answering tasks where the final answer is selected based on probability of answer choices. On the other hand, for models requiring reasoning, the method of answer extraction plays a critical role. Our research reveals that the performance of reasoning models and their final answer distributions are highly sensitive to the answer extraction algorithm employed. In order to mitigate this, we propose a basic framework: Answer Regeneration. The method uses an additional model inference, providing the prior input and output prefaced by the prompt "Answer:". The final answer is then selected or extracted from the regenerated output. We show that this extraction-rule-agnostic approach exhibits improved performance and enhanced robustness. Furthermore, we have applied this framework to general math problems and open-ended question answering tasks. Our analysis and this framework could offer a more reliable results for model evaluation.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 13 of its 23 references that are in the map. First posted on arXiv 16 Oct 2025.
Builds on
13 of 23 references.
Closest ancestors · 2
Same direction, newest first
- 2025Qwen3 Technical Report14 May
- 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluations14 Apr
Other references · 11
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
- 2024Measuring short-form factuality in large language modelscited by 345
- 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
- 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
- 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 132
- 2022Holistic Evaluation of Language Modelscited by 115
- 2024Efficient multi-prompt evaluation of LLMscited by 72
- 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026ABCD: All Biases Come Disguised19 Feb 2026
- 2026Do Instruction-Tuned Models Always Perform Better Than Base Models? Evidence from Math and Domain-Shifted Benchmarks19 Jan 2026
- 2026HyperCLOVA X 32B Think3 Jan 2026
Similar, not cited
Close in wording, no citation either way
- 2024Refining Answer Distributions for Improved Large Language Model Reasoning
- 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoning
- 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning
- 2025The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding?
- 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debate
- 2023Getting MoRE out of Mixture of Language Model Reasoning Experts
- 2023Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLMs