mlsanityData to 7 Oct 2026

Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs

arXiv 2412.16974 ↗First posted 22 Dec 2024cs.AI cs.CLcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Refusals - instances where large language models (LLMs) decline or fail to fully execute user instructions - are crucial for both AI safety and AI capabilities and the reduction of hallucinations in particular. These behaviors are learned during post-training, especially in instruction fine-tuning (IFT) and reinforcement learning from human feedback (RLHF). However, existing taxonomies and evaluation datasets for refusals are inadequate, often focusing solely on should-not-related (instead of cannot-related) categories, and lacking tools for auditing refusal content in black-box LLM outputs. We present a comprehensive framework for classifying LLM refusals: (a) a taxonomy of 16 refusal categories, (b) a human-annotated dataset of over 8,600 instances from publicly available IFT and RLHF datasets, (c) a synthetic dataset with 8,000 examples for each refusal category, and (d) classifiers trained for refusal classification. Our work enables precise auditing of refusal behaviors in black-box LLMs and automatic analyses of refusal patterns in large IFT and RLHF datasets. This facilitates the strategic adjustment of LLM refusals, contributing to the development of more safe and reliable LLMs.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 44 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

40 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Rule Based Rewards for Language Model Safety2 Nov
  2. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal20 Jun
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
24 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  6. 2023A Comprehensive Overview of Large Language Modelscited by 639
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  10. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  11. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  12. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  13. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  14. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  15. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  16. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  17. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 106
  18. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  19. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 55
  20. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 51
  21. 2023Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledgecited by 32
  22. 2024Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanationscited by 30
  23. 2023I'm Afraid I Can't Do That: Predicting Prompt Refusal in Black-Box Generative Language Modelscited by 6
  24. 2023Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025SAFEx: Analyzing Vulnerabilities of MoE-Based LLMs via Stable Safety-critical Expert Identificationcited by 14
  2. 2025What Large Language Models Do Not Talk About: An Empirical Study of Moderation and Censorship Practicescited by 11
  3. 2025Line of Duty: Evaluating LLM Self-Knowledge via Consistency in Feasibility Boundariescited by 7
  4. 2026Blind Refusal: Language Models Refuse to Help Users Evade Unjust, Absurd, and Illegitimate Rulescited by 2
  5. 2026From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directionscited by 1
  6. 2026RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Promptscited by 1
  7. 2025Position Paper: Rethinking Privacy in RL for Sequential Decision-making in the Age of LLMscited by 0
  8. 2025State-Dependent Refusal and Learned Incapacity in RLHF-Aligned Language Modelscited by 0
  9. 2026PAST2HARM: A Simple Adaptive Past Tense Attack for Jailbreaking Multimodal AIcited by 0
  10. 2026You Shouldn't Have Asked: A Pragmatics-Inspired Taxonomy for Evaluating LLM Refusalscited by 0
  11. 2026CompOrca: Corpus-Scale Compliance Labelling of Instruction-Tuning Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding Refusal in Language Models with Sparse Autoencoders
  2. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
  3. 2026Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks
  4. 2025SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models
  5. 2026Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMs
  6. 2025Think Before Refusal : Triggering Safety Reflection in LLMs to Mitigate False Refusal Behavior
  7. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models
  8. 2026Decomposing Refusal Steering in Mixture-of-Experts Models