mlsanityData to 7 Oct 2026

Exploring the Hidden Reasoning Process of Large Language Models by Misleading Them

arXiv 2503.16401 ↗First posted 20 Mar 2025cs.LGcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have been able to perform various forms of reasoning tasks in a wide range of scenarios, but are they truly engaging in task abstraction and rule-based reasoning beyond mere memorization? To answer this question, we propose a novel experimental approach, Misleading Fine-Tuning (MisFT), to examine whether LLMs perform abstract reasoning by altering their original understanding of fundamental rules. In particular, by constructing datasets with math expressions or logical formulas that contradict correct principles, we fine-tune the model to learn those contradictory rules and assess its generalization ability on unseen test domains. Through a series of experiments, we find that current LLMs are capable of applying contradictory rules to solve practical math word problems and natural language reasoning tasks, implying the presence of an internal mechanism in LLMs that abstracts before reasoning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 50 of its 73 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

50 of 73 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners16 Jun
  2. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models23 Apr
  3. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  4. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond16 Jun
  5. 2023Large Language Models Are Not Strong Abstract Reasoners31 May
  6. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-47 Apr

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023Mistral 7Bcited by 3,065
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
32 more
  1. 2018World Modelscited by 1,444
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  3. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  4. 2022A Survey on In-context Learningcited by 717
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  6. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  7. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  8. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  9. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  10. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  11. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  12. 2023Mathematical Capabilities of ChatGPTcited by 263
  13. 2022Data Contamination: From Memorization to Exploitationcited by 234
  14. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  15. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  16. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  17. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  18. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  19. 2019Counterfactual Story Reasoning and Generationcited by 87
  20. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  21. 2022Limitations of Language Models in Arithmetic and Symbolic Inductioncited by 71
  22. 2024Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelscited by 67
  23. 2020Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoningcited by 65
  24. 2023On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?cited by 47
  25. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 43
  26. 2024ACPBench: Reasoning about Action, Change, and Planningcited by 37
  27. 2021CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Modelscited by 33
  28. 2023Counterfactual reasoning: Testing language models' understanding of hypothetical scenarioscited by 27
  29. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  30. 2020SemEval-2020 Task 5: Counterfactual Recognitioncited by 12
  31. 2025When Do Neural Networks Learn World Models?cited by 8
  32. 2025RuozhiBench: Evaluating LLMs with Logical Fallacies and Misleading Premisescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinkingcited by 4
  2. 2026Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidencecited by 1
  3. 2026Reconciling In-Context and In-Weight Learning via Dual Representation Space Encodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance
  2. 2025Large Language Models and Mathematical Reasoning Failures
  3. 2024On Memorization of Large Language Models in Logical Reasoning
  4. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  5. 2025Exploring LLM Reasoning Through Controlled Prompt Variations
  6. 2025Reasoning Capabilities and Invariability of Large Language Models
  7. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  8. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges