# Self-Training Meets Consistency: Improving LLMs' Reasoning with Consistency-Driven Rationale Evaluation

arXiv: https://arxiv.org/abs/2411.06387 · HTML: https://mlsanity.com/paper/2411.06387/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 12 of its 50 references that are in the map.
- First posted on arXiv 10 Nov 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Synthetic data and self-improvement for LLMs / [Synthetic reasoning data and bootstrapping](https://mlsanity.com/synth/synthetic-reasoning-data-and-bootstrapping/)

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [Iterative Reasoning Preference Optimization](https://mlsanity.com/paper/2404.19733/) (2024)
- [How Likely Do LLMs with CoT Mimic Human Reasoning?](https://mlsanity.com/paper/2402.16048/) (2024)
- [LLMs May Perform MCQA by Selecting the Least Incorrect Option](https://mlsanity.com/paper/2402.01349/) (2024)
- [Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards](https://mlsanity.com/paper/2404.10346/) (2024)
- [Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning](https://mlsanity.com/paper/2401.13986/) (2024)
- [TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness](https://mlsanity.com/paper/2402.12545/) (2024)
- [Abstract Meaning Representation-Based Logic-Driven Data Augmentation for Logical Reasoning](https://mlsanity.com/paper/2305.12599/) (2023)

## Built on by (most cited first, 6 of 6)

- [AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners](https://mlsanity.com/paper/2505.16322/) (2025)
- [Self-Policy Distillation via Capability-Selective Subspace Projection](https://mlsanity.com/paper/2605.22675/) (2026)
- [TSUBASA: Improving Long-Horizon Personalization via Evolving Memory and Self-Learning with Context Distillation](https://mlsanity.com/paper/2604.07894/) (2026)
- [Skill Weaving: Efficient LLM Improvement via Modular Skillpacks](https://mlsanity.com/paper/2605.22205/) (2026)
- [PRISMA: Preference-Reinforced Self-Training Approach for Interpretable Emotionally Intelligent Negotiation Dialogues](https://mlsanity.com/paper/2604.18354/) (2026)
- [Evidence-Consistent Generative Detection under Scenario-Level Distribution Shift](https://mlsanity.com/paper/2608.21043/) (2026)

## Abstract (arXiv)

Self-training approach for large language models (LLMs) improves reasoning abilities by training the models on their self-generated rationales. Previous approaches have labeled rationales that produce correct answers for a given question as appropriate for training. However, a single measure risks misjudging rationale quality, leading the models to learn flawed reasoning patterns. To address this issue, we propose CREST (Consistency-driven Rationale Evaluation for Self-Training), a self-training framework that further evaluates each rationale through follow-up questions and leverages this evaluation to guide its training. Specifically, we introduce two methods: (1) filtering out rationales that frequently result in incorrect answers on follow-up questions and (2) preference learning based on mixed preferences from rationale evaluation results of both original and follow-up questions. Experiments on three question-answering datasets using open LLMs show that CREST not only improves the logical robustness and correctness of rationales but also improves reasoning abilities compared to previous self-training approaches.