# AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Models

arXiv: https://arxiv.org/abs/2505.00147 · HTML: https://mlsanity.com/paper/2505.00147/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [Active Example Selection for In-Context Learning](https://mlsanity.com/paper/2211.04486/) (2022)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws](https://mlsanity.com/paper/2404.05405/) (2024)
- [BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling](https://mlsanity.com/paper/2406.00832/) (2024)
- [Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse](https://mlsanity.com/paper/2410.21333/) (2024)
- [Faithful Logical Reasoning via Symbolic Chain-of-Thought](https://mlsanity.com/paper/2405.18357/) (2024)
- [Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving](https://mlsanity.com/paper/2405.12205/) (2024)
- [UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation](https://mlsanity.com/paper/2303.08518/) (2023)
- [On the Relation between Sensitivity and Accuracy in In-context Learning](https://mlsanity.com/paper/2209.07661/) (2022)
- [Skill-Based Few-Shot Selection for In-Context Learning](https://mlsanity.com/paper/2305.14210/) (2023)
- [AutoBencher: Towards Declarative Benchmark Construction](https://mlsanity.com/paper/2407.08351/) (2024)
- [Adaptive Testing of Computer Vision Models](https://mlsanity.com/paper/2212.02774/) (2022)
- [Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning](https://mlsanity.com/paper/2408.14774/) (2024)
- [EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Trees](https://mlsanity.com/paper/2503.08893/) (2025)
- [Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills](https://mlsanity.com/paper/2503.05641/) (2025)
- [Metadata Conditioning Accelerates Language Model Pre-training](https://mlsanity.com/paper/2501.01956/) (2025)
- [Can Large Language Models Act as Symbolic Reasoners?](https://mlsanity.com/paper/2410.21490/) (2024)

## Built on by (most cited first, 5 of 5)

- [Metacognitive Reuse: Turning Recurring LLM Reasoning Into Concise Behaviors](https://mlsanity.com/paper/2509.13237/) (2025)
- [Skill-Targeted Adaptive Training](https://mlsanity.com/paper/2510.10023/) (2025)
- [Error-Aware Curriculum Learning for Biomedical Relation Classification](https://mlsanity.com/paper/2507.14374/) (2025)
- [Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning](https://mlsanity.com/paper/2608.05139/) (2026)
- [CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes](https://mlsanity.com/paper/2608.27455/) (2026)

## Abstract (arXiv)

In-context learning (ICL) allows a language model to improve its problem-solving capability when provided with suitable information in context. Since the choice of in-context information can be determined based on the problem itself, in-context learning is analogous to human learning from teachers in a classroom. Recent works (Didolkar et al., 2024a; 2024b) show that ICL performance can be improved by leveraging a frontier large language model's (LLM) ability to predict required skills to solve a problem, popularly referred to as an LLM's metacognition, and using the recommended skills to construct necessary in-context examples. While this skill-based strategy boosts ICL performance in larger models, its gains on small language models (SLMs) have been minimal, highlighting a performance gap in ICL capabilities. We investigate this gap and show that skill-based prompting can hurt SLM performance on easy questions by introducing unnecessary information, akin to cognitive overload. To address this, we introduce AdaptMI, an adaptive approach to selecting skill-based in-context Math Instructions for SLMs. Inspired by cognitive load theory from human pedagogy, our method only introduces skill-based examples when the model performs poorly. We further propose AdaptMI+, which adds examples targeted to the specific skills missing from the model's responses. On 5-shot evaluations across popular math benchmarks and five SLMs (1B--7B; Qwen, Llama), AdaptMI+ improves accuracy by up to 6% over naive skill-based strategies.