# Influence Functions for Efficient Data Selection in Reasoning

arXiv: https://arxiv.org/abs/2510.06108 · HTML: https://mlsanity.com/paper/2510.06108/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 22 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models](https://mlsanity.com/paper/2405.20541/) (2024)
- [NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions](https://mlsanity.com/paper/2502.13124/) (2025)
- [Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models](https://mlsanity.com/paper/2411.12580/) (2024)
- [Large-Scale Data Selection for Instruction Tuning](https://mlsanity.com/paper/2503.01807/) (2025)
- [Efficient Data Selection at Scale via Influence Distillation](https://mlsanity.com/paper/2505.19051/) (2025)
- [Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages](https://mlsanity.com/paper/2501.14002/) (2025)
- [Compute-Constrained Data Selection](https://mlsanity.com/paper/2410.16208/) (2024)
- [Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoning](https://mlsanity.com/paper/2505.17266/) (2025)
- [Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling](https://mlsanity.com/paper/2501.13779/) (2025)

## Built on by (most cited first, 10 of 10)

- [Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment](https://mlsanity.com/paper/2601.14249/) (2026)
- [AIR: Post-training Data Selection for Reasoning via Attention Head Influence](https://mlsanity.com/paper/2512.13279/) (2025)
- [PRISM: Preference-Aware Influence Function Based Data Selection Method for Efficient Fine-Tuning](https://mlsanity.com/paper/2605.21422/) (2026)
- [When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection](https://mlsanity.com/paper/2605.26872/) (2026)
- [When Sample Selection Bias Precipitates Model Collapse](https://mlsanity.com/paper/2606.13732/) (2026)
- [DRIFT: Data Selection for LLM Instruction Tuning via On-Policy Attribution](https://mlsanity.com/paper/2606.18307/) (2026)
- [Towards a Data-Parameter Correspondence for LLMs: A Preliminary Discussion](https://mlsanity.com/paper/2604.17384/) (2026)
- [SafeLens: Deliberate and Efficient Video Guardrails with Fast-and-Slow Screening](https://mlsanity.com/paper/2605.17610/) (2026)
- [From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignment](https://mlsanity.com/paper/2605.21558/) (2026)
- [PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning](https://mlsanity.com/paper/2607.18199/) (2026)

## Abstract (arXiv)

Fine-tuning large language models (LLMs) on chain-of-thought (CoT) data shows that a small amount of high-quality data can outperform massive datasets. Yet, what constitutes "quality" remains ill-defined. Existing reasoning methods rely on indirect heuristics such as problem difficulty or trace length, while instruction-tuning has explored a broader range of automated selection strategies, but rarely in the context of reasoning. We propose to define reasoning data quality using influence functions, which measure the causal effect of individual CoT examples on downstream accuracy, and introduce influence-based pruning, which consistently outperforms perplexity and embedding-based baselines on math reasoning within a model family.