# Are Aligned Large Language Models Still Misaligned?

arXiv: https://arxiv.org/abs/2602.11305 · HTML: https://mlsanity.com/paper/2602.11305/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs](https://mlsanity.com/paper/2503.08688/) (2025)
- [Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?](https://mlsanity.com/paper/2501.15463/) (2025)
- [An Evaluation of Cultural Value Alignment in LLM](https://mlsanity.com/paper/2504.08863/) (2025)
- [Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency](https://mlsanity.com/paper/2506.17209/) (2025)
- [ReIFE: Re-evaluating Instruction-Following Evaluation](https://mlsanity.com/paper/2410.07069/) (2024)
- [ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMs](https://mlsanity.com/paper/2409.09586/) (2024)
- [Too Helpful, Too Harmless, Too Honest or Just Right?](https://mlsanity.com/paper/2509.08486/) (2025)
- [An Investigation into Value Misalignment in LLM-Generated Texts for Cultural Heritage](https://mlsanity.com/paper/2501.02039/) (2025)
- [Societal Alignment Frameworks Can Improve LLM Alignment](https://mlsanity.com/paper/2503.00069/) (2025)
- [From Narrow Unlearning to Emergent Misalignment: Causes, Consequences, and Containment in LLMs](https://mlsanity.com/paper/2511.14017/) (2025)
- [Right Prediction, Wrong Reasoning: Uncovering LLM Misalignment in RA Disease Diagnosis](https://mlsanity.com/paper/2504.06581/) (2025)
- [A Survey of LLM Prompt Datasets: Taxonomy, Linguistic Patterns, and Practical Uses](https://mlsanity.com/paper/2510.09316/) (2025)
- [Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL](https://mlsanity.com/paper/2510.06092/) (2025)

## Abstract (arXiv)

Misalignment in Large Language Models (LLMs) arises when model behavior diverges from human expectations and fails to simultaneously satisfy safety, value, and cultural dimensions, which must co-occur in real-world settings to solve a real-world query. Existing misalignment benchmarks-such as INSECURE CODE (safety-centric), VALUEACTIONLENS (value-centric), and CULTURALHERITAGE (culture centric)-rely on evaluating misalignment along individual dimensions, preventing simultaneous evaluation. To address this gap, we introduce Mis-Align Bench, a unified benchmark for analyzing misalignment across safety, value, and cultural dimensions. First we constructs SAVACU, an English misaligned-aligned dataset of 382,424 samples spanning 112 domains (or labels), by reclassifying prompts from the LLM-PROMPT-DATASET via taxonomy into 14 safety domains, 56 value domains, and 42 cultural domains using Mistral-7B-Instruct-v0.3, and expanding low-resource domains via Llama-3.1-8B-Instruct with SimHash-based fingerprint to avoid deduplication. Furthermore, we pairs prompts with misaligned and aligned responses via two-stage rejection sampling to enforce quality. Second we benchmarks general-purpose, fine-tuned, and open-weight LLMs, enabling systematic evaluation of misalignment under three dimensions. Empirically, single-dimension models achieve high Coverage (upto 97.6%) but incur False Failure Rate >50% and lower Alignment Score (63%-66%) under joint conditions.