Data to 14 Oct 2026

Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability

arXiv 2411.19943 ↗First posted 29 Nov 2024cs.CL cs.AI cs.LGcited by 83

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Mathematical reasoning tasks pose significant challenges for large language models (LLMs) because they require precise logical deduction and sequence analysis. In this work, we introduce the concept of critical tokens -- elements within reasoning trajectories that significantly influence incorrect outcomes. We present a novel framework for identifying these tokens through rollout sampling and demonstrate their substantial divergence from traditional error tokens. Through extensive experiments on datasets such as GSM8K and MATH500, we show that identifying and replacing critical tokens significantly improves model accuracy. We propose an efficient methodology for pinpointing these tokens in large-scale datasets using contrastive estimation and extend this framework to enhance model training processes with direct preference optimization (DPO). Experimental results on GSM8K and MATH500 benchmarks with the widely used models Llama-3 (8B and 70B) and Deepseek-math (7B) demonstrate the effectiveness of the proposed approach, cDPO. Our results underscore the potential of leveraging critical tokens to reduce errors in reasoning tasks, advancing the development of AI systems capable of robust logical deduction. Our code, annotated datasets, and trained models are available at https://github.com/chenzhiling9954/Critical-Tokens-Matter to support and encourage future research in this promising field.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 50 of its 53 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

50 of 53 references.

Closest ancestors · 1

Same direction, newest first

2023Contrastive Decoding Improves Reasoning in Large Language Models17 Sep

Other references · 49

2022Training language models to follow instructions with human feedbackcited by 15,971
2017Proximal Policy Optimization Algorithmscited by 15,215
2024The Llama 3 Herd of Modelscited by 14,715
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2017Deep reinforcement learning from human preferencescited by 3,912
2023Let's Verify Step by Stepcited by 3,789
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720

37 more

2019Fine-Tuning Language Models from Human Preferencescited by 2,150
2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
2024Iterative Reasoning Preference Optimizationcited by 246
2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
2024Token-level Direct Preference Optimizationcited by 135
2024Rho-1: Not All Tokens Are What You Needcited by 133
2024Direct Preference Optimization with an Offsetcited by 122
2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 79
2023Evaluating Language Models for Mathematics through Interactionscited by 70
2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspectivecited by 69
2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
2024Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Factscited by 36
2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
2018Adversarial Contrastive Estimationcited by 23
2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrastcited by 19
2024Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparatorscited by 19
2021Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learningcited by 18
2024Safety Alignment of Large Language Models via Contrasting Safe and Harmful Distributionscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 38; 2026: 43.

83 papers.

Sort:

By area:

2024Phi-4 Technical Reportcited by 711
2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 225
2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
2025Agentic Reinforced Policy Optimizationcited by 172
2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 57
2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 51
2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 47
2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 40
2025Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Modelscited by 39

38 more

2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 39
2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 39
2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 36
2025Reinforcing Video Reasoning with Focused Thinkingcited by 31
2025Estimating LLM Uncertainty with Evidencecited by 26
2026On the Position Bias of On-Policy Distillationcited by 26
2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 25
2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 22
2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMscited by 20
2025AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Modelscited by 17
2025Group Critical-token Policy Optimization for Autoregressive Image Generationcited by 17
2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 16
2025Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Pointscited by 14
2025Token-Importance Guided Direct Preference Optimizationcited by 14
2025Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critiquecited by 13
2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
2025SynAdapt: Learning Adaptive Reasoning in Large Language Models via Synthetic Continuous Chain-of-Thoughtcited by 13
2024LLM2: Let Large Language Models Harness System 2 Reasoningcited by 12
2025PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoningcited by 11
2025Teaching Your Models to Understand Code via Focal Preference Alignmentcited by 10
2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMscited by 10
2025DISC: Dynamic Decomposition Improves LLM Inference Scalingcited by 9
2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoningcited by 9
2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Modelscited by 9
2026RelayLLM: Efficient Reasoning via Collaborative Decodingcited by 8
2026TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Taskscited by 8
2026Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoningcited by 6
2025RAST: Reasoning Activation in LLMs via Small-model Transfercited by 5
2026Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal Statescited by 5
2026Linear Dynamics in the RLVR Training of Large Language Modelscited by 4
2026Training-Trajectory-Aware Token Selectioncited by 4
2025Self-Reflective Generation at Test Timecited by 3
2025LightReasoner: Can Small Language Models Teach Large Language Models Reasoning?cited by 3
2026APPO: Attention-guided Perception Policy Optimization for Video Reasoningcited by 3
2026Rethinking Personalization in Large Language Models at the Token Levelcited by 3
2026Learning Adaptive LLM Decodingcited by 3
2026Select to Think: Unlocking SLM Potential with Local Sufficiencycited by 3
2026How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMscited by 3

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way