mlsanityData to 7 Oct 2026

Premise Order Matters in Reasoning with Large Language Models

arXiv 2402.08939 ↗First posted 14 Feb 2024cs.AI cs.CLcited by 71

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models (LLMs) have accomplished remarkable reasoning performance in various domains. However, in the domain of reasoning tasks, we discover a frailty: LLMs are surprisingly brittle to the ordering of the premises, despite the fact that such ordering does not alter the underlying task. In particular, we observe that LLMs achieve the best performance when the premise order aligns with the context required in intermediate reasoning steps. For example, in deductive reasoning tasks, presenting the premises in the same order as the ground truth proof in the prompt (as opposed to random ordering) drastically increases the model's accuracy. We first examine the effect of premise ordering on deductive reasoning on a variety of LLMs, and our evaluation shows that permuting the premise order can cause a performance drop of over 30%. In addition, we release the benchmark R-GSM, based on GSM8K, to examine the ordering effect for mathematical problem-solving, and we again observe a significant drop in accuracy, relative to the original GSM8K benchmark.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 39 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

30 of 39 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
18 more
  1. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  2. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  3. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  4. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  5. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  6. 2022Thinking Fast and Slow in Large Language Modelscited by 136
  7. 2023Teaching Arithmetic to Small Transformerscited by 121
  8. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  9. 2022On the Paradox of Learning to Reason from Datacited by 93
  10. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  11. 2016DeepMath - Deep Sequence Models for Premise Selectioncited by 80
  12. 2020UnNatural Language Inferencecited by 68
  13. 2023Large Language Models can Learn Rulescited by 66
  14. 2017Premise Selection for Theorem Proving by Deep Graph Embeddingcited by 44
  15. 2022Word Order Does Matter (And Shuffled Language Models Know It)cited by 32
  16. 2024LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Modelscited by 23
  17. 2023Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Textcited by 19
  18. 2019Kendall Tau Sequence Distance: Extending Kendall Tau from Ranks to Sequencescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 29; 2026: 20.

71 papers.

Sort:

By area:

  1. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 208
  2. 2025Hierarchical Reasoning Modelcited by 167
  3. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  4. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 83
  5. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  6. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  7. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 56
  8. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  9. 2024Break the Chain: Large Language Models Can be Shortcut Reasonerscited by 32
  10. 2024Can LLMs Solve longer Math Word Problems Better?cited by 22
  11. 2024To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planningcited by 21
  12. 2024Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Modelscited by 20
38 more
  1. 2026Large Language Model Reasoning Failurescited by 20
  2. 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
  3. 2024A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoningcited by 15
  4. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Modelscited by 13
  5. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 13
  6. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  7. 2025Order Matters: Investigate the Position Bias in Multi-constraint Instruction Followingcited by 12
  8. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  9. 2024A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?cited by 9
  10. 2025HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputscited by 9
  11. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  12. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Datacited by 8
  13. 2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
  14. 2025Advancing the Scientific Method with Large Language Models: From Hypothesis to Discoverycited by 7
  15. 2024Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problemscited by 6
  16. 2024Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approachcited by 6
  17. 2025Training Large Language Models To Reason In Parallel With Global Forking Tokenscited by 6
  18. 2025Representation Consistency for Accurate and Coherent LLM Answer Aggregationcited by 5
  19. 2025Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inferencecited by 5
  20. 2024Can Language Models Pretend Solvers? Logic Code Simulation with LLMscited by 4
  21. 2025Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judgecited by 4
  22. 2025Fragile Preferences: A Deep Dive Into Order Effects in Large Language Modelscited by 4
  23. 2025Your Language Model May Think Too Rigidly: Achieving Reasoning Consistency with Symmetry-Enhanced Trainingcited by 3
  24. 2025Weak-to-Strong GraphRAG: Aligning Weak Retrievers with Large Language Models for Graph-based Retrieval Augmented Generationcited by 3
  25. 2025SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligencecited by 3
  26. 2026Beyond representational alignment with brain-guided language models for robust reasoningcited by 3
  27. 2025DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Modelscited by 2
  28. 2025The Impossibility of Inverse Permutation Learning in Transformer Modelscited by 2
  29. 2026LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scalingcited by 2
  30. 2026Gaze Heads: How VLMs Look at What They Describecited by 2
  31. 2026Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Modelscited by 2
  32. 2024First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoningcited by 1
  33. 2024Models Can and Should Embrace the Communicative Nature of Human-Generated Mathcited by 1
  34. 2024Mediating Modes of Thought: LLM's for design scriptingcited by 1
  35. 2025DIMSUM: Discourse in Mathematical Reasoning as a Supervision Modulecited by 1
  36. 2025Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Modelscited by 1
  37. 2025Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbationscited by 1
  38. 2026GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagramscited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DeduCE: Deductive Consistency as a Framework to Evaluate LLM Reasoning
  2. 2025Exploring LLM Reasoning Through Controlled Prompt Variations
  3. 2024Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Models
  4. 2024Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning
  5. 2024The Role of Deductive and Inductive Reasoning in Large Language Models
  6. 2025THiNK: Can Large Language Models Think-aloud?
  7. 2023Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
  8. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving