# Thinking with Nothinking Calibration: A New In-Context Learning Paradigm in Reasoning Large Language Models

arXiv: https://arxiv.org/abs/2508.03363 · HTML: https://mlsanity.com/paper/2508.03363/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [Reasoning Models Can Be Effective Without Thinking](https://mlsanity.com/paper/2504.09858/) (2025)
- [CoT-Valve: Length-Compressible Chain-of-Thought Tuning](https://mlsanity.com/paper/2502.09601/) (2025)
- [Distilling System 2 into System 1](https://mlsanity.com/paper/2407.06023/) (2024)
- [The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks](https://mlsanity.com/paper/2502.08235/) (2025)
- [AdaptThink: Reasoning Models Can Learn When to Think](https://mlsanity.com/paper/2505.13417/) (2025)
- [Fast Best-of-N Decoding via Speculative Rejection](https://mlsanity.com/paper/2410.20290/) (2024)
- [Reward-Guided Speculative Decoding for Efficient LLM Reasoning](https://mlsanity.com/paper/2501.19324/) (2025)
- [AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning](https://mlsanity.com/paper/2505.11896/) (2025)
- [Symbol tuning improves in-context learning in language models](https://mlsanity.com/paper/2305.08298/) (2023)
- [Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decoding](https://mlsanity.com/paper/2503.01422/) (2025)
- [When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs](https://mlsanity.com/paper/2505.11423/) (2025)
- [Why Larger Language Models Do In-context Learning Differently?](https://mlsanity.com/paper/2405.19592/) (2024)
- [Using In-Context Learning to Improve Dialogue Safety](https://mlsanity.com/paper/2302.00871/) (2023)
- [On the generalization of language models from in-context learning and finetuning: a controlled study](https://mlsanity.com/paper/2505.00661/) (2025)
- [Z1: Efficient Test-time Scaling with Code](https://mlsanity.com/paper/2504.00810/) (2025)
- [Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMs](https://mlsanity.com/paper/2503.10084/) (2025)
- [Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking](https://mlsanity.com/paper/2503.19602/) (2025)
- [FastMCTS: A Simple Sampling Strategy for Data Synthesis](https://mlsanity.com/paper/2502.11476/) (2025)
- [AdapThink: Adaptive Thinking Preferences for Reasoning Language Model](https://mlsanity.com/paper/2506.18237/) (2025)
- [Demonstration Selection for In-Context Learning via Reinforcement Learning](https://mlsanity.com/paper/2412.03966/) (2024)
- [Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrieval](https://mlsanity.com/paper/2411.04006/) (2024)
- [Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions](https://mlsanity.com/paper/2406.10878/) (2024)

## Built on by (most cited first, 4 of 4)

- [FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline](https://mlsanity.com/paper/2510.06800/) (2025)
- [Inference-Time Conformal Reasoning with Valid Factuality Control for Large Language Models](https://mlsanity.com/paper/2606.08831/) (2026)
- [Fast, Slow, and Tool-augmented Thinking for LLMs: A Review](https://mlsanity.com/paper/2508.12265/) (2025)
- [Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposure](https://mlsanity.com/paper/2609.27763/) (2026)

## Abstract (arXiv)

Reasoning large language models (RLLMs) have recently demonstrated remarkable capabilities through structured and multi-step reasoning. While prior research has primarily focused on improving their training and inference strategies, their potential for in-context learning (ICL) remains largely underexplored. To fill this gap, we propose Thinking with Nothinking Calibration (JointThinking), a new ICL paradigm that prompts the model to generate two answers in parallel: one in Thinking mode and the other in Nothinking mode. A second round of Thinking is triggered only when the two initial responses are inconsistent, using a single prompt with two different answers. Extensive experiments across multiple reasoning benchmarks demonstrate that JointThinking significantly outperforms few-shot chain-of-thought (CoT), thinking twice and majority voting. Moreover, it achieves comparable in-distribution performance to training-based SOTA reasoning method, while substantially outperforming on out-of-distribution tasks. We further conduct a systematic analysis of the calibration mechanism, showing the importance of structural thinking diversity and the benefits of consistency check. Additionally, we observe that the performance gap between actual and ideal reasoning narrows as model size increases in the second thinking, indicating the strong scalability of our approach. Finally, we discuss current limitations and outline promising directions for future ICL research in RLLMs.