mlsanityData to 7 Oct 2026

Online Data Selection Is Implicit Alignment

arXiv 2607.07023 ↗First posted 8 Jul 2026cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Supervised fine-tuning (SFT) is often treated as a capability-adaptation step, while alignment is attributed to later preference optimization or reinforcement learning. This separation is incomplete: when examples are scored and kept online during fine-tuning, the choice of which data to train on already changes the model's behavioral preferences. We study online data selection as an implicit alignment mechanism. Given the same base model, optimizer, and selected-token budget, we compare random, loss-based, quality-based, and diversity-based online selectors and measure the behavioral drift they induce without any preference optimization. The proposed evaluation tracks helpfulness, refusal rate, verbosity, truthfulness, sycophancy, calibration, and jailbreak robustness, together with diagnostics for which behavioral modes are over-represented in the selected data. We formalize online selection as a reweighted SFT objective whose weights define an implicit preference over response styles and safety postures, so that an online scorer plays the role usually assigned to a reward model. This view predicts that high-scoring data can systematically favor longer, more assertive, more compliant, or more refusal-prone behaviors depending on how the online score is defined. Empirically, selectors that are statistically indistinguishable in task accuracy diverge sharply in refusal rate, verbosity, and sycophancy, and we show that the direction of the shift is predictable from the attribute mixture of the selected data. We introduce Alignment Drift Auditing (ADA), a controlled protocol for quantifying selection-induced behavioral movement, and Alignment-Aware Selection (AAS), a diagnostic online selector that retains data efficiency while constraining drift along safety and style axes.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 58 of its 70 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

58 of 70 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  3. 2022Training language models to follow instructions with human feedback4 Mar
  4. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 54

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2016Learning without Forgettingcited by 2,465
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
38 more
  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  11. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  12. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  13. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  14. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  15. 2023Textbooks Are All You Needcited by 561
  16. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  17. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  18. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  19. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  20. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  21. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  22. 2023AI Alignment: A Comprehensive Surveycited by 256
  23. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  24. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  25. 2024A Survey on Data Selection for Language Modelscited by 230
  26. 2024Mixture of LoRA Expertscited by 179
  27. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  28. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  29. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  30. 2015Online Batch Selection for Faster Training of Neural Networkscited by 121
  31. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  32. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  33. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  34. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  35. 2025Decouple and Orthogonalize: A Data-Free Framework for LoRA Mergingcited by 17
  36. 2024Energy-based Automated Model Evaluationcited by 16
  37. 2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 14
  38. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 10

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage
  2. 2025The Three Regimes of Offline-to-Online Reinforcement Learning
  3. 2025From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learning
  4. 2025Offline Behavioral Data Selection
  5. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment
  6. 2024Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization
  7. 2025The Reward Model Selection Crisis in Personalized Alignment
  8. 2026Online Self-Weighted Fine-Tuning