mlsanityData to 14 Oct 2026

Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models

arXiv 2410.05269 ↗First posted 7 Oct 2024cs.CL cs.AI cs.LGcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Data is a crucial element in large language model (LLM) alignment. Recent studies have explored using LLMs for efficient data collection. However, LLM-generated data often suffers from quality issues, with underrepresented or absent aspects and low-quality datapoints. To address these problems, we propose Data Advisor, an enhanced LLM-based method for generating data that takes into account the characteristics of the desired dataset. Starting from a set of pre-defined principles in hand, Data Advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation accordingly. Data Advisor can be easily integrated into existing data generation methods to enhance data quality and coverage. Experiments on safety alignment of three representative LLMs (i.e., Mistral, Llama2, and Falcon) demonstrate the effectiveness of Data Advisor in enhancing model safety against various fine-grained safety issues without sacrificing model utility.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 42 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

32 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!19 Feb
  2. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  3. 2023Safer-Instruct: Aligning Language Models with Automated Preference Data15 Nov
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
16 more
  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  3. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  4. 2023The Falcon Series of Open Language Modelscited by 535
  5. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  6. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  7. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  8. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  9. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 163
  10. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  11. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 83
  12. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  13. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Datacited by 24
  14. 2024SelectLLM: Can LLMs Select Important Instructions to Annotate?cited by 16
  15. 2024GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Constructioncited by 15
  16. 2024From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verificationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  2. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
  3. 2025Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilitiescited by 20
  4. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 14
  5. 2025DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Modelscited by 5
  6. 2025Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creationcited by 3
  7. 2025Learning to Extract Context for Context-Aware LLM Inferencecited by 2
  8. 2026Alignment-Weighted DPO: A principled reasoning approach to improve safety alignmentcited by 2
  9. 2025MEUV: Achieving Fine-Grained Capability Activation in Large Language Models via Mutually Exclusive Unlock Vectorscited by 0
  10. 2025Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Modelscited by 0
  11. 2026Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxiescited by 0
  12. 2026APEX: Automated Prompt Engineering eXpert with Dynamic Data Selectioncited by 0
1 more
  1. 2026Faking Good and Faking Bad in LLMs: Response Distortion Across Dark Triad Personality Traitscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunch
  2. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety
  3. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
  4. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  5. 2026DataShield: Uncovering Risky Fine-Tuning Data Across LLMs Through Consensus Subspace Alignment
  6. 2026PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models
  7. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations
  8. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning