mlsanityData to 14 Oct 2026

TaP: A Taxonomy-Guided Framework for Automated and Scalable Preference Data Generation

arXiv 2506.23979 ↗First posted 30 Jun 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Conducting supervised and preference fine-tuning of large language models (LLMs) requires high-quality datasets to improve their ability to follow instructions and align with human preferences and values. However, constructing such datasets is resource-intensive, and most publicly available datasets are in English. To address these challenges, we propose the underlineTaxonomy-Guided underlinePreference Data Generation (TaP) framework for automated, scalable preference dataset construction across languages. TaP uses a structured taxonomy to provide fine-grained control over dataset composition, ensuring diversity and broad coverage. We use TaP-generated datasets to perform supervised and preference fine-tuning on multiple LLMs. Experimental results demonstrate that LLMs trained on TaP-generated datasets outperform those trained on existing open-source datasets. Remarkably, LLMs trained on TaP-generated datasets outperform models trained on an open-source dataset that is 180× larger.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

35 of 47 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  7. 2023LIMA: Less Is More for Alignmentcited by 1,100
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  10. 2023Large Language Modelscited by 780
  11. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
23 more
  1. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  2. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  5. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  7. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  8. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  9. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  10. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  11. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  12. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  13. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 115
  14. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  15. 2023Human Feedback is not Gold Standardcited by 83
  16. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 79
  17. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
  18. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52
  19. 2024COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuningcited by 45
  20. 2019Translationese as a Language in "Multilingual" NMTcited by 36
  21. 2023Chinese Open Instruction Generalist: A Preliminary Releasecited by 35
  22. 2023The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarizationcited by 5
  23. 2025From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Groundingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
  4. 2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset
  5. 2025Refine-n-Judge: Curating High-Quality Preference Chains for LLM-Fine-Tuning
  6. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignment
  7. 2026Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
  8. 2024Multi-Response Preference Optimization with Augmented Ranking Dataset