mlsanityData to 14 Oct 2026

Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training

arXiv 2608.16926 ↗First posted 5 Aug 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Data selection in supervised fine-tuning aims to select a small set of effective samples from large-scale candidate data, reducing training cost while preserving model performance. However, existing methods usually treat data value as a relatively static property, and pay limited attention to the compatibility between data and the capability distribution of the target model. To address this issue, we propose Data-DPO, a target model-oriented SFT data selection method. Data-DPO observes the local training feedback of the target model on different samples through one-step probing, transforms activation differences among samples into pairwise data preferences, and trains a lightweight reward model to learn target-model-aware data preferences. In the final selection stage, Data-DPO further combines target model preference, external quality scores, and marginal diversity to construct a more stable and effective training subset. Experimental results on Vision-Flan and LLaVA-CoT show that Data-DPO consistently outperforms existing data selection baselines under multiple data budgets and stably surpasses full data training performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 45 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

36 of 45 references.

References · 36

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2025Qwen3-VL Technical Reportcited by 2,793
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
24 more
  1. 2016A Diagram Is Worth A Dozen Imagescited by 804
  2. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  3. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  5. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  6. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  7. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 286
  8. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
  9. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  10. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  11. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuningcited by 73
  12. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 69
  13. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 61
  14. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52
  15. 2024ICONS: Influence Consensus for Vision-Language Data Selectioncited by 29
  16. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 21
  17. 2025FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gaincited by 15
  18. 2025T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuningcited by 13
  19. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  20. 2025CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimizationcited by 8
  21. 2025Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectoriescited by 7
  22. 2026ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuningcited by 5
  23. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottleneckscited by 4
  24. 2025Learning from the Best, Differently: A Diversity-Driven Rethinking on Data Selectioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PRISM: Preference-Aware Influence Function Based Data Selection Method for Efficient Fine-Tuning
  2. 2026Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training
  3. 2024COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Framework
  4. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimization
  5. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  6. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  7. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  8. 2024Intermediate direct preference optimization