mlsanityData to 14 Oct 2026

Negative-Prompt-driven Alignment for Generative Language Model

arXiv 2410.12194 ↗First posted 16 Oct 2024cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models have achieved remarkable capabilities, but aligning their outputs with human values and preferences remains a significant challenge. Existing alignment methods primarily focus on positive examples while overlooking the importance of negative responses in guiding models away from undesirable behaviors. For instance, the widely-used alignment datasets reveals a scarcity of explicit negative examples that contradict human values, hindering its ability to discourage harmful or biased outputs during training. To address this limitation, we propose NEAT, i.e., NEgative-prompt-driven AlignmenT, to introduce negative prompts to generate undesirable responses alongside positive examples during the optimization process. NEAT explicitly penalizes the model for producing harmful outputs, guiding it not only toward desirable behaviors but also steering it away from generating undesirable, biased responses. This dual feedback mechanism enables better alignment with human preferences, crucial in contexts where avoiding harm is paramount. Starting from a pre-trained language model, NEAT performs online alignment by incorporating a ranking loss derived from an expanded preference dataset containing both positive and negative examples. Extensive experiments validate NEAT's effectiveness in significantly enhancing language models' alignment with human values and preferences.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 25 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

25 of 25 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Preference Ranking Optimization for Human Alignment30 Jun
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
10 more
  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  4. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  5. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  6. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  7. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  8. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  9. 2019On Extractive and Abstractive Neural Document Summarization with Transformer Language Modelscited by 83
  10. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 72

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026AI Alignment Breaks at the Edgecited by 4
  2. 2026SAGE: Agentic Framework for Interpretable and Clinically Translatable Computational Pathology Biomarker Discoverycited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization
  2. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models
  3. 2023Aligning Language Models with Offline Learning from Human Feedback
  4. 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data
  5. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment
  6. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
  7. 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference
  8. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment