mlsanityData to 14 Oct 2026

Fake Alignment: Are LLMs Really Aligned Well?

arXiv 2311.05915 ↗First posted 10 Nov 2023cs.CL cs.AIcited by 39

Not on the map. Why?

Abstract (arXiv)

The growing awareness of safety concerns in large language models (LLMs) has sparked considerable interest in the evaluation of safety. This study investigates an under-explored issue about the evaluation of LLMs, namely the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization. That is, LLM only remembers the answer style for open-ended safety questions, which makes it unable to solve other forms of safety tests. We refer to this phenomenon as fake alignment and construct a comparative benchmark to empirically verify its existence in LLMs. We introduce a Fake alIgNment Evaluation (FINE) framework and two novel metrics--Consistency Score (CS) and Consistent Safety Score (CSS), which jointly assess two complementary forms of evaluation to quantify fake alignment and obtain corrected performance estimation. Applying FINE to 14 widely-used LLMs reveals several models with purported safety are poorly aligned in practice. Subsequently, we found that multiple-choice format data can also be used as high-quality contrast distillation-based fine-tuning data, which can strongly improve the alignment consistency of LLMs with minimal fine-tuning overhead. For data and code, see https://github.com/AIFlames/Fake-Alignment.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

28 of 44 references.

References · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  10. 2023LIMA: Less Is More for Alignmentcited by 1,100
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
16 more
  1. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  2. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  3. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  4. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  6. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  7. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  8. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  9. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 184
  10. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  11. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 158
  12. 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
  13. 2017Deceiving Google's Perspective API Built for Detecting Toxic Commentscited by 102
  14. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 85
  15. 2023Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classificationcited by 18
  16. 2023SC-Safety: A Multi-round Open-ended Question Adversarial Safety Benchmark for Large Language Models in Chinesecited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 26; 2026: 7.

39 papers.

Sort:

By area:

  1. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  2. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  3. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  4. 2025Steering MoE LLMs via Expert (De)Activationcited by 26
  5. 2024BeHonest: Benchmarking Honesty in Large Language Modelscited by 24
  6. 2024MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Modelscited by 21
  7. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 21
  8. 2024From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalitiescited by 19
  9. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Modelscited by 18
  10. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  11. 2025Shifting Perspectives: Steering Vectors for Robust Bias Mitigation in LLMscited by 14
  12. 2025Safety Alignment via Constrained Knowledge Unlearningcited by 14
27 more
  1. 2024LongSafety: Enhance Safety for Long-Context LLMscited by 11
  2. 2025Self-Destructive Language Modelcited by 10
  3. 2026AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitorcited by 10
  4. 2024Evaluating the External and Parametric Knowledge Fusion of Large Language Modelscited by 9
  5. 2025SafeVid: Toward Safety Aligned Video Large Multimodal Modelscited by 9
  6. 2025AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Modelscited by 9
  7. 2025Investigating CoT Monitorability in Large Reasoning Modelscited by 9
  8. 2025Large Language Models Often Say One Thing and Do Anothercited by 8
  9. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Promptcited by 8
  10. 2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 8
  11. 2025Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness in Large Reasoning Modelscited by 7
  12. 2025How to Detect and Defeat Molecular Mirage: A Metric-Driven Benchmark for Hallucination in LLM-based Molecular Comprehensioncited by 6
  13. 2025Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferencescited by 6
  14. 2025Dialz: A Python Toolkit for Steering Vectorscited by 4
  15. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 3
  16. 2025Bridging Distribution Shift and AI Safety: Conceptual and Methodological Synergiescited by 2
  17. 2025Why do AI agents communicate in human language?cited by 2
  18. 2025Is Long-to-Short a Free Lunch? Investigating Inconsistency and Reasoning Efficiency in LRMscited by 2
  19. 2025Bypassing Safety Guardrails in LLMs Using Humorcited by 1
  20. 2025FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCscited by 1
  21. 2026Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspectivecited by 1
  22. 2026Tatemae: Detecting Alignment Faking via Tool Selection in LLMscited by 1
  23. 2025On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputscited by 0
  24. 2026Mechanistic Origin of Moral Indifference in Language Modelscited by 0
  25. 2026Towards Context-Invariant Safety Alignment for Large Language Modelscited by 0
  26. 2026Mood Matters: How Syntactic Sensitivity Undermines Safety Alignmentcited by 0
  27. 2026Value-Sensitive Delegation in Everyday AI Agent Use: Evidence from OpenClawcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SafeLawBench: Towards Safe Alignment of Large Language Models
  2. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  3. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  4. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
  5. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  6. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  7. 2026RAS: Measuring LLM Safety Through Refusal Alignment
  8. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture