# Small Language Model as Data Prospector for Large Language Model

arXiv: https://arxiv.org/abs/2412.09990 · HTML: https://mlsanity.com/paper/2412.09990/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers](https://mlsanity.com/paper/2212.10559/) (2022)
- [M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning](https://mlsanity.com/paper/2306.04387/) (2023)
- [Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases](https://mlsanity.com/paper/2303.14742/) (2023)
- [Instruction Mining: Instruction Data Selection for Tuning Large Language Models](https://mlsanity.com/paper/2307.06290/) (2023)
- [COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning](https://mlsanity.com/paper/2403.18058/) (2024)
- [One-Shot Learning as Instruction Data Prospector for Large Language Models](https://mlsanity.com/paper/2312.10302/) (2023)
- [Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data](https://mlsanity.com/paper/2304.01196/) (2023)

## Built on by (most cited first, 2 of 2)

- [Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations](https://mlsanity.com/paper/2508.03550/) (2025)
- [Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practice](https://mlsanity.com/paper/2512.24503/) (2025)

## Abstract (arXiv)

The quality of instruction data directly affects the performance of fine-tuned Large Language Models (LLMs). Previously, citeli2023one proposed textttNUGGETS, which identifies and selects high-quality quality data from a large dataset by identifying those individual instruction examples that can significantly improve the performance of different tasks after being learnt as one-shot instances. In this work, we propose textttSuperNUGGETS, an improved variant of textttNUGGETS optimised for efficiency and performance. Our textttSuperNUGGETS uses a small language model (SLM) instead of a large language model (LLM) to filter the data for outstanding one-shot instances and refines the predefined set of tests. The experimental results show that the performance of textttSuperNUGGETS only decreases by 1-2% compared to textttNUGGETS, but the efficiency can be increased by a factor of 58. Compared to the original textttNUGGETS, our textttSuperNUGGETS has a higher utility value due to the significantly lower resource consumption.