mlsanityData to 7 Oct 2026

The Art of Saying No: Contextual Noncompliance in Language Models

arXiv 2407.12043 ↗First posted 2 Jul 2024cs.CL cs.AI cs.HCcited by 100

Not on the map. Why?

Cited by 23 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Chat-based language models are designed to be helpful, yet they should not comply with every user request. While most existing work primarily focuses on refusal of "unsafe" queries, we posit that the scope of noncompliance should be broadened. We introduce a comprehensive taxonomy of contextual noncompliance describing when and how models should not comply with user requests. Our taxonomy spans a wide range of categories including incomplete, unsupported, indeterminate, and humanizing requests (in addition to unsafe requests). To test noncompliance capabilities of language models, we use this taxonomy to develop a new evaluation suite of 1000 noncompliance prompts. We find that most existing models show significantly high compliance rates in certain previously understudied categories with models like GPT-4 incorrectly complying with as many as 30% of requests. To address these gaps, we explore different training strategies using a synthetically-generated training set of requests and expected noncompliant responses. Our experiments demonstrate that while direct finetuning of instruction-tuned models can lead to both over-refusal and a decline in general capabilities, using parameter efficient methods like low rank adapters helps to strike a good balance between appropriate noncompliance and other capabilities.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 82 of its 125 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

82 of 125 references.

References · 82

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  10. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  12. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  13. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  14. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  15. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  16. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  17. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  18. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  19. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  20. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  21. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  22. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  23. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  24. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  25. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  26. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  27. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  28. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  29. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  30. 2024LoRA Learns Less and Forgets Lesscited by 313
  31. 2020Calibration of Pre-trained Transformerscited by 307
  32. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  33. 2020Selective Question Answering under Domain Shiftcited by 228
  34. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  35. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  36. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  37. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  38. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 40; 2026: 47.

100 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  3. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  4. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  5. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  6. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  7. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  8. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  9. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  10. 2025Large-Scale Data Selection for Instruction Tuningcited by 34
  11. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencycited by 25
  12. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 22
38 more
  1. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  2. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 18
  3. 2025Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusalcited by 17
  4. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  5. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 14
  6. 2025Epistemic Alignment: A Mediating Framework for User-LLM Knowledge Deliverycited by 14
  7. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  8. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learningcited by 14
  9. 2026Knowing When to Abstain: Medical LLMs Under Clinical Uncertaintycited by 13
  10. 2024Acceptable Use Policies for Foundation Modelscited by 12
  11. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 12
  12. 2026Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RLcited by 11
  13. 2024SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behaviorcited by 10
  14. 2024SafeWorld: Geo-Diverse Safety Alignmentcited by 10
  15. 2025TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Storiescited by 10
  16. 2026Over-Searching in Search-Augmented Large Language Modelscited by 10
  17. 2024Fine-Tuning Large Language Models to Appropriately Abstain with Semantic Entropycited by 9
  18. 2025SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questionscited by 9
  19. 2025Don't Throw Away Your Pretrained Modelcited by 9
  20. 2026Agentic Abstention: Do Agents Know When to Stop Instead of Act?cited by 9
  21. 2025Discovering Forbidden Topics in Language Modelscited by 8
  22. 2025RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Modelscited by 8
  23. 2025Nile-Chat: Egyptian Language Models for Arabic and Latin Scriptscited by 7
  24. 2025Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Predictioncited by 7
  25. 2026DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretrainingcited by 7
  26. 2025ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Datacited by 6
  27. 2025DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collectionscited by 6
  28. 2025Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewardscited by 6
  29. 2026Among Us: Measuring and Mitigating Malicious Contributions in Model Collaboration Systemscited by 6
  30. 2025Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Modelscited by 5
  31. 2025Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinationscited by 5
  32. 2025Reasoning Up the Instruction Ladder for Controllable Language Modelscited by 5
  33. 2025AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learningcited by 5
  34. 2026MoCo: A One-Stop Shop for Model Collaboration Researchcited by 5
  35. 2025Sparse Neurons Carry Strong Signals of Question Ambiguity in LLMscited by 4
  36. 2025Characterizing Selective Refusal Bias in Large Language Modelscited by 4
  37. 2026The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agentscited by 4
  38. 2024Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editingcited by 3

50 most cited of 100 All 100 (JSON)