# ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes

arXiv: https://arxiv.org/abs/2610.15336 · HTML: https://mlsanity.com/paper/2610.15336/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Data mixtures and data selection, one of 8 directions in LLM pre-training data and scaling laws.
- Builds on 15 of its 41 references that are in the map.
- First posted on arXiv 11 Oct 2026.

## Placed in

- LLM pre-training data and scaling laws / [Data mixtures and data selection](https://mlsanity.com/pretrain/data-mixtures-and-data-selection/)

## Builds on (most cited first, 15 of 15)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [EmbeddingGemma: Powerful and Lightweight Text Representations](https://mlsanity.com/paper/2509.20354/) (2025)
- [Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark](https://mlsanity.com/paper/2402.11592/) (2024)
- [SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning](https://mlsanity.com/paper/2404.18239/) (2024)
- [Large-Scale Data Selection for Instruction Tuning](https://mlsanity.com/paper/2503.01807/) (2025)
- [G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translation](https://mlsanity.com/paper/2405.12915/) (2024)
- [When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale](https://mlsanity.com/paper/2309.04564/) (2023)
- [Z0-Inf: Zeroth Order Approximation for Data Influence](https://mlsanity.com/paper/2510.11832/) (2025)
- [Daunce: Data Attribution through Uncertainty Estimation](https://mlsanity.com/paper/2505.23223/) (2025)

## Abstract (arXiv)

Influence functions enable principled data selection for large language models (e.g., identifying influential training samples for model editing, debugging, or efficient fine-tuning). However, scaling IFs to LLMs is hindered by the intractability of billion-scale Hessian matrices and the prohibitive memory overhead of backpropagation (BP). While many classical IF approximations reduce Hessian complexity, they still rely on BP for gradient computation, requiring massive storage for intermediate activations that impedes practical data selection at scale. To address this, we propose ZOIF, a zeroth-order framework for influence estimation that operates entirely via forward passes, enabling memory-efficient data selection under resource constraints. ZOIF decomposes influence estimation into three BP-free stages: (1) approximating the average proxy gradient via zeroth-order estimation; (2) reformulating and solving the inverse Hessian-vector product as an auxiliary zeroth-order optimization to bypass explicit inversion; and (3) deriving per-sample influence scores via a forward finite difference along the previously computed parameter update direction. Experiments show that ZOIF selects data subsets with downstream utility competitive with, and in some cases exceeding, the evaluated baselines, yet at substantially lower peak memory cost and with comparable overall computation time.