mlsanityData to 14 Oct 2026

Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?

arXiv 2507.15707 ↗First posted 21 Jul 2025cs.CL cs.AIcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) have been evaluated using diverse question types, e.g., multiple-choice, true/false, and short/long answers. This study answers an unexplored question about the impact of different question types on LLM accuracy on reasoning tasks. We investigate the performance of five LLMs on three different types of questions using quantitative and deductive reasoning tasks. The performance metrics include accuracy in the reasoning steps and choosing the final answer. Key Findings: (1) Significant differences exist in LLM performance across different question types. (2) Reasoning accuracy does not necessarily correlate with the final selection accuracy. (3) The number of options and the choice of words, influence LLM performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 26 of its 35 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

26 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoning25 May

Other references · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
13 more
  1. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  3. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  4. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  5. 2018Gender Bias in Coreference Resolutioncited by 442
  6. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  7. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  8. 2020Transformers as Soft Reasoners over Languagecited by 270
  9. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  10. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  11. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 80
  12. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 71
  13. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 69

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025When Facts Change: Probing LLMs on Evolving Knowledge with evolveQAcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating the Meta- and Object-Level Reasoning of Large Language Models for Question Answering
  2. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  3. 2024Confidence in the Reasoning of Large Language Models
  4. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  5. 2023Evaluating the Deductive Competence of Large Language Models
  6. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  7. 2025Reasoning Models are Test Exploiters: Rethinking Multiple-Choice
  8. 2025Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong