mlsanityData to 19 Oct 2026

ICLHerd: Herding-Based Data Selection for LLMs with In-Context Learning as a Proxy

arXiv 2610.27142 ↗First posted 19 Oct 2026cs.CL cs.AI cs.LGcited by 0

On the map: LLM pre-training data and scaling laws → Data mixtures and data selection

Abstract (arXiv)

Data selection plays a central role in modern large language model training, as the quality of training data strongly influences downstream performance. A common approach is to curate a development set that encodes the desired skills (e.g., reasoning or multilinguality) and select data that are similar to them. Existing methods compute similarity using cosine distance over either the model's hidden states or gradients. However, we observe that this max-similarity strategy induces a mode-seeking bias: the selected data clusters tightly around the development samples. To address this, we propose ICLHerd, a data selection framework that leverages the von Mises-Fisher kernel to balance mode-seeking with mode-averaging. We further introduce in-context learning as a validation proxy, enabling efficient hyperparameter search over the balance trade-off. Results show that ICLHerd outperforms existing LLM data selection algorithms, highlighting the importance of balancing mode-seeking and mode-averaging.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Data mixtures and data selection, one of 8 directions in LLM pre-training data and scaling laws. Builds on 38 of its 54 references that are in the map. First posted on arXiv 19 Oct 2026.

Builds on

38 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024A Survey on Data Selection for Language Models26 Feb
  2. 2024QuRating: Selecting High-Quality Data for Training Language Models15 Feb
  3. 2023Data Selection for Language Models via Importance Resampling6 Feb

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,760
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,450
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,308
  4. 2025Qwen3 Technical Reportcited by 9,258
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,787
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,067
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,505
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,312
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,201
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,758
  11. 2024A Survey on LLM-as-a-Judgecited by 1,475
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,335
23 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,219
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,153
  3. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,110
  4. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 919
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 730
  6. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 553
  7. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 553
  8. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 429
  9. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 396
  10. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 375
  11. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 352
  12. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 318
  13. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 240
  14. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 170
  15. 2024How to Train Data-Efficient LLMscited by 101
  16. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 87
  17. 2025The Best Instruction-Tuning Data are Those That Fitcited by 55
  18. 2025Large-Scale Data Selection for Instruction Tuningcited by 40
  19. 2023AlpaGasus: Training A Better Alpaca with Fewer Datacited by 34
  20. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 31
  21. 2025Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilitiescited by 24
  22. 2024Chasing Random: Instruction Selection Strategies Fail to Generalizecited by 14
  23. 2024Compute-Constrained Data Selectioncited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training
  2. 2026Is One-Shot In-Context Learning Helpful for Data Selection in Task-Specific Fine-Tuning of Multimodal LLMs?
  3. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning
  4. 2025BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
  5. 2025LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection
  6. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
  7. 2026Data Mixing for Large Language Models Pretraining: A Survey and Outlook
  8. 2026Aligning the Query Space: Greedy Information Projection for Language Model Data Selection