mlsanityData to 14 Oct 2026

First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning

arXiv 2406.16078 ↗First posted 23 Jun 2024cs.CLcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Multi-step reasoning instruction, such as chain-of-thought prompting, is widely adopted to explore better language models (LMs) performance. We report on the systematic strategy that LMs employ in such a multi-step reasoning process. Our controlled experiments reveal that LMs rely more heavily on heuristics, such as lexical overlap, in the earlier stages of reasoning, where more reasoning steps remain to reach a goal. Conversely, their reliance on heuristics decreases as LMs progress closer to the final answer through multiple reasoning steps. This suggests that LMs can backtrack only a limited number of future steps and dynamically combine heuristic strategies with rationale ones in tasks involving multi-step reasoning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 9 of its 30 references that are in the map. First posted on arXiv 23 Jun 2024.

Builds on

9 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023PaLM 2 Technical Report17 May
  2. 2022Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review24 Feb

Other references · 7

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  6. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 85
  7. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 72

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29

Similar, not cited

Close in wording, no citation either way

  1. 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models
  2. 2025Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?
  3. 2023Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models
  4. 2024Multi-Step Reasoning with Large Language Models, a Survey
  5. 2025Reasoning Capabilities of Large Language Models on Dynamic Tasks
  6. 2024Plan of Thoughts: Heuristic-Guided Problem Solving with Large Language Models
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  8. 2025Reasoning LLMs are Wandering Solution Explorers