# Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning

arXiv: https://arxiv.org/abs/2412.09906 · HTML: https://mlsanity.com/paper/2412.09906/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 0 of its 33 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Built on by (most cited first, 11 of 11)

- [Guiding Reasoning in Small Language Models with LLM Assistance](https://mlsanity.com/paper/2504.09923/) (2025)
- [NavigScene: Bridging Local Perception and Global Navigation for Beyond-Visual-Range Autonomous Driving](https://mlsanity.com/paper/2507.05227/) (2025)
- [CoT2Align: Cross-Chain of Thought Distillation via Optimal Transport Alignment for Language Models with Different Tokenizers](https://mlsanity.com/paper/2502.16806/) (2025)
- [Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More](https://mlsanity.com/paper/2503.10542/) (2025)
- [TinyGraphEstimator: Adapting Lightweight Language Models for Graph Structure Inference](https://mlsanity.com/paper/2510.08808/) (2025)
- [Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering](https://mlsanity.com/paper/2509.10249/) (2025)
- [Error-Aware Curriculum Learning for Biomedical Relation Classification](https://mlsanity.com/paper/2507.14374/) (2025)
- [WRAVAL -- WRiting Assist eVALuation](https://mlsanity.com/paper/2601.03268/) (2025)
- [SLM Finetuning for Natural Language to Domain Specific Code Generation in Production](https://mlsanity.com/paper/2604.09952/) (2026)
- [Structural Rationale Distillation via Reasoning Space Compression](https://mlsanity.com/paper/2605.07139/) (2026)
- [STEMMA: An Adversarial Multi-Agent Framework for Evaluating Self-Identity Consistency in LLMs](https://mlsanity.com/paper/2608.08164/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks. Advances in prompt engineering and fine-tuning techniques have further enhanced their ability to address complex reasoning challenges. However, these advanced capabilities are often exclusive to models exceeding 100 billion parameters. Although Chain-of-Thought (CoT) fine-tuning methods have been explored for smaller models (under 10 billion parameters), they typically depend on extensive CoT training data, which can introduce inconsistencies and limit effectiveness in low-data settings. To overcome these limitations, this paper introduce a new reasoning strategy Solution Guidance (SG) and a plug-and-play training paradigm Solution-Guidance Fine-Tuning (SGFT) for enhancing the reasoning capabilities of small language models. SG focuses on problem understanding and decomposition at the semantic and logical levels, rather than specific computations, which can effectively improve the SLMs' generalization and reasoning abilities. With only a small amount of SG training data, SGFT can fine-tune a SLM to produce accurate problem-solving guidances, which can then be flexibly fed to any SLM as prompts, enabling it to generate correct answers directly. Experimental results demonstrate that our method significantly improves the performance of SLMs on various reasoning tasks, enhancing both their practicality and efficiency within resource-constrained environments.