mlsanityData to 7 Oct 2026

Let the Target Select for Itself: Data Selection via Target-Aligned Paths

arXiv 2605.09404 ↗First posted 10 May 2026cs.LG cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Targeted data selection seeks training examples from a candidate pool that improve downstream task performance. While trajectory-based selectors effectively guide this process, existing approaches typically construct reference states by warming up on the candidate pool itself, thereby inheriting pool-dependent distributional biases. To address this coupling, we introduce Target-Aligned Candidate Selection (TACS), which constructs a short, capacity-constrained reference path using a compact target-validation proxy. Candidates are ranked by their normalized loss reduction along this trajectory, requiring only forward passes and allowing the path to be reused across distinct pools for a fixed model-target pair. Empirically, TACS achieves competitive downstream performance across controlled logistic, vision, and NLP benchmarks, while slashing selection compute by up to 69% and candidate storage to less than 10 MB in large-scale instruction tuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 62 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

43 of 62 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  9. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  10. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  11. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  12. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
31 more
  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  3. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  4. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  5. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  6. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  7. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  8. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 259
  9. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  10. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  11. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  12. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  13. 2020The Early Phase of Neural Network Trainingcited by 114
  14. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 76
  15. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  16. 2025Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuningcited by 39
  17. 2024Rethinking Data Shapley for Data Selection Tasks: Misleads and Meritscited by 29
  18. 2024CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-trainingcited by 20
  19. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 17
  20. 2024ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuningcited by 10
  21. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learningcited by 10
  22. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 9
  23. 2024Influential Language Data Selection via Gradient Trajectory Pursuitcited by 7
  24. 2026GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometrycited by 7
  25. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
  26. 2025Train on Validation (ToV): Fast data selection with applications to fine-tuningcited by 3
  27. 2025BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretrainingcited by 3
  28. 2026Linearization Explains Fine-Tuning in Large Language Modelscited by 2
  29. 2026Aligning the Query Space: Greedy Information Projection for Language Model Data Selectioncited by 1
  30. 2026Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuningcited by 1
  31. 2026Target-Oriented Pretraining Data Selection via Neuron-Activated Graphcited by 1

Built on by

Not cited yet.