# Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignment

arXiv: https://arxiv.org/abs/2504.09802 · HTML: https://mlsanity.com/paper/2504.09802/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [KTO: Model Alignment as Prospect Theoretic Optimization](https://mlsanity.com/paper/2402.01306/) (2024)
- [Self-Play Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2405.00675/) (2024)
- [Advancing LLM Reasoning Generalists with Preference Trees](https://mlsanity.com/paper/2404.02078/) (2024)
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022)
- [Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions](https://mlsanity.com/paper/2411.14405/) (2024)
- [Explanations from Large Language Models Make Small Reasoners Better](https://mlsanity.com/paper/2210.06726/) (2022)
- [β-DPO: Direct Preference Optimization with Dynamic β](https://mlsanity.com/paper/2407.08639/) (2024)
- [Noise Contrastive Alignment of Language Models with Explicit Rewards](https://mlsanity.com/paper/2402.05369/) (2024)
- [Distillation Scaling Laws](https://mlsanity.com/paper/2502.08606/) (2025)
- [Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling](https://mlsanity.com/paper/2408.16737/) (2024)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification](https://mlsanity.com/paper/2410.05318/) (2024)
- [Explaining Question Answering Models through Text Generation](https://mlsanity.com/paper/2004.05569/) (2020)
- [Mixed Distillation Helps Smaller Language Model Better Reasoning](https://mlsanity.com/paper/2312.10730/) (2023)
- [A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directions](https://mlsanity.com/paper/2504.09100/) (2025)

## Built on by (most cited first, 7 of 7)

- [Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotations](https://mlsanity.com/paper/2505.10937/) (2025)
- [Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs](https://mlsanity.com/paper/2509.14257/) (2025)
- [Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards](https://mlsanity.com/paper/2601.22511/) (2026)
- [A Mechanistic View of Authority Hierarchy in LLM Sycophancy](https://mlsanity.com/paper/2607.00415/) (2026)
- [AgenticQwen: Training Small Agentic Language Models with Dual Data Flywheels for Industrial-Scale Tool Use](https://mlsanity.com/paper/2604.21590/) (2026)
- [EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2505.20888/) (2025)
- [Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning](https://mlsanity.com/paper/2606.04466/) (2026)

## Abstract (arXiv)

The reasoning capabilities of large reasoning models (LRMs), such as OpenAI's o1 and DeepSeek-R1, have seen substantial advancements through deep thinking. However, these enhancements come with significant resource demands, underscoring the need for training effective small reasoning models. A critical challenge is that small models possess different reasoning capacities and cognitive trajectories compared with their larger counterparts. Hence, directly distilling chain-of-thought (CoT) rationales from large LRMs to smaller ones can sometimes be ineffective and often requires a substantial amount of annotated data. In this paper, we first introduce a novel Critique-Rethink-Verify (CRV) system, designed for training smaller yet powerful LRMs. Our CRV system consists of multiple LLM agents, each specializing in unique tasks: (i) critiquing the CoT rationales according to the cognitive capabilities of smaller models, (ii) rethinking and refining these CoTs based on the critiques, and (iii) verifying the correctness of the refined results. Building on the CRV system, we further propose the Cognitive Preference Optimization (CogPO) algorithm to continuously enhance the reasoning abilities of smaller models by aligning their reasoning processes with their cognitive capacities. Comprehensive evaluations on challenging reasoning benchmarks demonstrate the efficacy of our CRV+CogPO framework, which outperforms other methods by a large margin.