mlsanityData to 7 Oct 2026

Suppressing Pink Elephants with Direct Principle Feedback

arXiv 2402.07896 ↗First posted 12 Feb 2024cs.CLcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Existing methods for controlling language models, such as RLHF and Constitutional AI, involve determining which LLM behaviors are desirable and training them into a language model. However, in many cases, it is desirable for LLMs to be controllable at inference time, so that they can be used in multiple contexts with diverse needs. We illustrate this with the Pink Elephant Problem: instructing an LLM to avoid discussing a certain entity (a ``Pink Elephant''), and instead discuss a preferred entity (``Grey Elephant''). We apply a novel simplification of Constitutional AI, Direct Principle Feedback, which skips the ranking of responses and uses DPO directly on critiques and revisions. Our results show that after DPF fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2 model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and performs as well as GPT-4 in on our curated test set assessing the Pink Elephant Problem.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 64 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

46 of 64 references.

References · 46

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
34 more
  1. 2023Let's Verify Step by Stepcited by 3,598
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2022Mass-Editing Memory in a Transformercited by 976
  9. 2023Steering Language Models With Activation Engineeringcited by 824
  10. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  12. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  13. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  14. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  15. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  16. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  17. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  18. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  19. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  20. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  21. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  22. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  23. 2023Fine-tuning Language Models for Factualitycited by 219
  24. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  25. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  26. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  27. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  28. 2023Shepherd: A Critic for Language Model Generationcited by 96
  29. 2023Stay on topic with Classifier-Free Guidancecited by 90
  30. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  31. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  32. 2023This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Modelscited by 18
  33. 2021Cut the CARP: Fishing for zero-shot story evaluationcited by 14
  34. 2022Robust Preference Learning for Storytelling via Contrastive Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 4; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  2. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  3. 2025Reinforcement Learning from Human Feedbackcited by 53
  4. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  5. 2024RLVF: Learning from Verbal Feedback without Overgeneralizationcited by 18
  6. 2024CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialoguescited by 15
  7. 2024Configurable Safety Tuning of Language Models with Synthetic Preference Datacited by 12
  8. 2024Self-Directed Synthetic Dialogues and Revisions Technical Reportcited by 7
  9. 2025Negation: A Pink Elephant in the Large Language Models' Room?cited by 7
  10. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  11. 2024MISR: Measuring Instrumental Self-Reasoning in Frontier Modelscited by 2
  12. 2026Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Failcited by 2
1 more
  1. 2025Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
  2. 2024Direct Language Model Alignment from Online AI Feedback
  3. 2023Specific versus General Principles for Constitutional AI
  4. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
  5. 2024Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller
  6. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B
  7. 2025How Effective Is Constitutional AI in Small LLMs? A Study on DeepSeek-R1 and Its Peers
  8. 2026FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents