mlsanityData to 14 Oct 2026

Unintended Harms of Value-Aligned LLMs: Psychological and Empirical Insights

arXiv 2506.06404 ↗First posted 6 Jun 2025cs.CL cs.AI cs.CYcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

The application scope of Large Language Models (LLMs) continues to expand, leading to increasing interest in personalized LLMs that align with human values. However, aligning these models with individual values raises significant safety concerns, as certain values may correlate with harmful information. In this paper, we identify specific safety risks associated with value-aligned LLMs and investigate the psychological principles behind these challenges. Our findings reveal two key insights. (1) Value-aligned LLMs are more prone to harmful behavior compared to non-fine-tuned models and exhibit slightly higher risks in traditional safety evaluations than other fine-tuned models. (2) These safety issues arise because value-aligned LLMs genuinely generate text according to the aligned values, which can amplify harmful outcomes. Using a dataset with detailed safety categories, we find significant correlations between value alignment and safety risks, supported by psychological hypotheses. This study offers insights into the "black box" of value alignment and proposes in-context alignment methods to enhance the safety of value-aligned LLMs.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 51 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

21 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 18

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  6. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  7. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  8. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  9. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  10. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  11. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  12. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
6 more
  1. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
  2. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 97
  3. 2019A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Datacited by 74
  4. 2023ROBBIE: Robust Bias Evaluation of Large Generative Language Modelscited by 53
  5. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 50
  6. 2023The Touch'e23-ValueEval Dataset for Identifying Human Values behind Argumentscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Human Values Matter: Investigating How Misalignment Shapes Collective Behaviors in LLM Agent Communitiescited by 4
  2. 2025IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimizationcited by 1
  3. 2026Value Alignment Tax: Measuring Value Trade-offs in LLM Alignmentcited by 1
  4. 2026Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructurecited by 1
  5. 2026AdaCultureSafe: Adaptive Cultural Safety Grounded by Cultural Knowledge in Large Language Modelscited by 0
  6. 2026How Value Induction Reshapes LLM Behaviourcited by 0
  7. 2026Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  2. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context
  3. 2025Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Users
  4. 2026Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks
  5. 2025Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior
  6. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
  7. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety
  8. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts