# On the Thinking-Language Modeling Gap in Large Language Models

arXiv: https://arxiv.org/abs/2505.12896 · HTML: https://mlsanity.com/paper/2505.12896/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 30 of its 74 references that are in the map.
- First posted on arXiv 19 May 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 30)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [Chain-of-Thought Reasoning Without Prompting](https://mlsanity.com/paper/2402.10200/) (2024)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [The pitfalls of next-token prediction](https://mlsanity.com/paper/2403.06963/) (2024)
- [Distilling System 2 into System 1](https://mlsanity.com/paper/2407.06023/) (2024)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [The Consciousness Prior](https://mlsanity.com/paper/1709.08568/) (2017)
- [Chain of Thoughtlessness? An Analysis of CoT in Planning](https://mlsanity.com/paper/2405.04776/) (2024)
- [System 2 Attention (is something you might need too)](https://mlsanity.com/paper/2311.11829/) (2023)
- [Premise Order Matters in Reasoning with Large Language Models](https://mlsanity.com/paper/2402.08939/) (2024)
- [Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models](https://mlsanity.com/paper/2406.02061/) (2024)
- [Contrastive Chain-of-Thought Prompting](https://mlsanity.com/paper/2311.09277/) (2023)
- [The Expressive Power of Transformers with Chain of Thought](https://mlsanity.com/paper/2310.07923/) (2023)
- [Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning](https://mlsanity.com/paper/2407.01687/) (2024)
- [Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities](https://mlsanity.com/paper/2406.02787/) (2024)
- [Prompting Fairness: Integrating Causality to Debias Large Language Models](https://mlsanity.com/paper/2403.08743/) (2024)
- [Strong hallucinations from negation and how to fix them](https://mlsanity.com/paper/2402.10543/) (2024)
- ["Let's Argue Both Sides": Argument Generation Can Force Small Models to Utilize Previously Inaccessible Reasoning Capabilities](https://mlsanity.com/paper/2410.12997/) (2024)

## Abstract (arXiv)

System 2 reasoning is one of the defining characteristics of intelligence, which requires slow and logical thinking. Human conducts System 2 reasoning via the language of thoughts that organizes the reasoning process as a causal sequence of mental language, or thoughts. Recently, it has been observed that System 2 reasoning can be elicited from Large Language Models (LLMs) pre-trained on large-scale natural languages. However, in this work, we show that there is a significant gap between the modeling of languages and thoughts. As language is primarily a tool for humans to share knowledge and thinking, modeling human language can easily absorb language biases into LLMs deviated from the chain of thoughts in minds. Furthermore, we show that the biases will mislead the eliciting of "thoughts" in LLMs to focus only on a biased part of the premise. To this end, we propose a new prompt technique termed Language-of-Thoughts (LoT) to demonstrate and alleviate this gap. Instead of directly eliciting the chain of thoughts from partial information, LoT instructs LLMs to adjust the order and token used for the expressions of all the relevant information. We show that the simple strategy significantly reduces the language modeling biases in LLMs and improves the performance of LLMs across a variety of reasoning tasks.