mlsanityData to 14 Oct 2026

Are Aligned Large Language Models Still Misaligned?

arXiv 2602.11305 ↗First posted 11 Feb 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Misalignment in Large Language Models (LLMs) arises when model behavior diverges from human expectations and fails to simultaneously satisfy safety, value, and cultural dimensions, which must co-occur in real-world settings to solve a real-world query. Existing misalignment benchmarks-such as INSECURE CODE (safety-centric), VALUEACTIONLENS (value-centric), and CULTURALHERITAGE (culture centric)-rely on evaluating misalignment along individual dimensions, preventing simultaneous evaluation. To address this gap, we introduce Mis-Align Bench, a unified benchmark for analyzing misalignment across safety, value, and cultural dimensions. First we constructs SAVACU, an English misaligned-aligned dataset of 382,424 samples spanning 112 domains (or labels), by reclassifying prompts from the LLM-PROMPT-DATASET via taxonomy into 14 safety domains, 56 value domains, and 42 cultural domains using Mistral-7B-Instruct-v0.3, and expanding low-resource domains via Llama-3.1-8B-Instruct with SimHash-based fingerprint to avoid deduplication. Furthermore, we pairs prompts with misaligned and aligned responses via two-stage rejection sampling to enforce quality. Second we benchmarks general-purpose, fine-tuned, and open-weight LLMs, enabling systematic evaluation of misalignment under three dimensions. Empirically, single-dimension models achieve high Coverage (upto 97.6%) but incur False Failure Rate >50% and lower Alignment Score (63%-66%) under joint conditions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 29 references that are in the map. First posted on arXiv 11 Feb 2026.

Builds on

19 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2021Ethical and social risks of harm from Language Modelscited by 966
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  5. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  6. 2025Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMscited by 54
  7. 2025Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?cited by 51
  8. 2025An Evaluation of Cultural Value Alignment in LLMcited by 28
  9. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencycited by 27
  10. 2024ReIFE: Re-evaluating Instruction-Following Evaluationcited by 22
  11. 2024ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMscited by 20
  12. 2025Too Helpful, Too Harmless, Too Honest or Just Right?cited by 9
6 more
  1. 2025An Investigation into Value Misalignment in LLM-Generated Texts for Cultural Heritagecited by 8
  2. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  3. 2025From Narrow Unlearning to Emergent Misalignment: Causes, Consequences, and Containment in LLMscited by 6
  4. 2025Right Prediction, Wrong Reasoning: Uncovering LLM Misalignment in RA Disease Diagnosiscited by 4
  5. 2025A Survey of LLM Prompt Datasets: Taxonomy, Linguistic Patterns, and Practical Usescited by 2
  6. 2025Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RLcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Can Large Language Models Make Everyone Happy?
  2. 2026AlignCultura: Towards Culturally Aligned Large Language Models?
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  4. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
  5. 2023CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models
  6. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
  7. 2026The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs
  8. 2024ABC Align: Large Language Model Alignment for Safety & Accuracy