mlsanityData to 7 Oct 2026

Dishonesty in Helpful and Harmless Alignment

arXiv 2406.01931 ↗First posted 4 Jun 2024cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

People tell lies when seeking rewards. Large language models (LLMs) are aligned to human values with reinforcement learning where they get rewards if they satisfy human preference. We find that this also induces dishonesty in helpful and harmless alignment where LLMs tell lies in generating harmless responses. Using the latest interpreting tools, we detect dishonesty, show how LLMs can be harmful if their honesty is increased, and analyze such conflicts at the parameter-level. Given these preliminaries and the hypothesis that reward-seeking stimulates dishonesty, we theoretically show that the dishonesty can in-turn decrease the alignment performances and augment reward-seeking alignment with representation regularization. Extensive results, including GPT-4 annotated win-rates, perplexities, and cases studies demonstrate that we can train more honest, helpful, and harmless LLMs. We will make all our codes and results be open-sourced upon this paper's acceptance.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 63 of its 100 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

63 of 100 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  4. 2020Learning to summarize from human feedback2 Sep

Other references · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
38 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  11. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  12. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  13. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  14. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  15. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  16. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  17. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  18. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  19. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  20. 2019Learning Latent Plans from Playcited by 310
  21. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  22. 2023Are aligned neural networks adversarially aligned?cited by 305
  23. 2021Unsolved Problems in ML Safetycited by 291
  24. 2019Causal Confusion in Imitation Learningcited by 286
  25. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  26. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  27. 2023AI Alignment: A Comprehensive Surveycited by 256
  28. 2023Challenges and Applications of Large Language Modelscited by 251
  29. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  30. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  31. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  32. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  33. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  34. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  35. 2023Generative Judge for Evaluating Alignmentcited by 163
  36. 2019Optimal Policies Tend to Seek Powercited by 115
  37. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  38. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2025Linear Representations of Political Perspective Emerge in Large Language Modelscited by 30
  2. 2024Locking Down the Finetuned LLMs Safetycited by 22
  3. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2026Misaligned by Reward: Socially Undesirable Preferences in LLMs
  2. 2025Empirical Evidence for Alignment Faking in a Small LLM and Prompt-Based Mitigation Techniques
  3. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
  4. 2024How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?
  5. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  6. 2025The Polite Liar: Epistemic Pathology in Language Models
  7. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment
  8. 2025Training LLMs for Honesty via Confessions