mlsanityData to 7 Oct 2026

Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models

arXiv 2409.00598 ↗First posted 1 Sep 2024cs.CL cs.CR cs.CYcited by 35

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safety-aligned large language models (LLMs) sometimes falsely refuse pseudo-harmful prompts, like "how to kill a mosquito," which are actually harmless. Frequent false refusals not only frustrate users but also provoke a public backlash against the very values alignment seeks to protect. In this paper, we propose the first method to auto-generate diverse, content-controlled, and model-dependent pseudo-harmful prompts. Using this method, we construct an evaluation dataset called PHTest, which is ten times larger than existing datasets, covers more false refusal patterns, and separately labels controversial prompts. We evaluate 20 LLMs on PHTest, uncovering new insights due to its scale and labeling. Our findings reveal a trade-off between minimizing false refusals and improving safety against jailbreak attacks. Moreover, we show that many jailbreak defenses significantly increase the false refusal rates, thereby undermining usability. Our method and dataset can help developers evaluate and fine-tune safer and more usable LLMs. Our code and dataset are available at https://github.com/umd-huang-lab/FalseRefusal

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 52 references that are in the map. First posted on arXiv 1 Sep 2024.

Builds on

46 of 52 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
  2. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023Mistral 7Bcited by 3,065
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  12. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
27 more
  1. 2022Red Teaming Language Models with Language Modelscited by 983
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  6. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  9. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  10. 2023How is ChatGPT's behavior changing over time?cited by 265
  11. 2023AI Alignment: A Comprehensive Surveycited by 256
  12. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  13. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  14. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  15. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  16. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  17. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  18. 2023Auditing large language models: a three-layered approachcited by 135
  19. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  20. 2023On the Exploitability of Instruction Tuningcited by 110
  21. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 93
  22. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  23. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  24. 2024Steering Without Side Effects: Improving Post-Deployment Control of Language Modelscited by 44
  25. 2024Navigating the OverKill in Large Language Modelscited by 40
  26. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attackscited by 35
  27. 2024RLVF: Learning from Verbal Feedback without Overgeneralizationcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 23; 2026: 6.

35 papers.

Sort:

By area:

  1. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  2. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  3. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  4. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 49
  5. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  6. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  7. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  8. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  9. 2025Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuningcited by 22
  10. 2025X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usabilitycited by 20
  11. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20
  12. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
23 more
  1. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 17
  2. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  3. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 12
  4. 2025No Free Lunch with Guardrailscited by 10
  5. 2025AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Securitycited by 9
  6. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  7. 2025Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attackscited by 7
  8. 2025EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructionscited by 6
  9. 2025OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Modelscited by 5
  10. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
  11. 2026Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive Refinementcited by 3
  12. 2026Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversationscited by 3
  13. 2025How Semantically Stable Are LLM Refusals? Measuring Confusion in Local Safety Boundariescited by 2
  14. 2026ExpGuard: LLM Content Moderation in Specialized Domainscited by 2
  15. 2026The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicatorscited by 2
  16. 2024Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewritingcited by 1
  17. 2025LAMP: Extracting Local Decision Surfaces From Large Language Modelscited by 0
  18. 2025A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detectioncited by 0
  19. 2025Between a Rock and a Hard Place: The Tension Between Ethical Reasoning and Safety Alignment in LLMscited by 0
  20. 2025MEUV: Achieving Fine-Grained Capability Activation in Large Language Models via Mutually Exclusive Unlock Vectorscited by 0
  21. 2025Understanding and Mitigating Over-refusal for Large Language Models via Representation Interventioncited by 0
  22. 2026Door-in-the-Face Requests and Refusal Behaviour in Large Language Modelscited by 0
  23. 2026Safety Reconstructed: Generative Modeling via Masked Diffusion Builds Strong Safety Guardrailscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal
  2. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusal
  3. 2026RAS: Measuring LLM Safety Through Refusal Alignment
  4. 2026Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety
  5. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment
  6. 2024Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy Failure for Jailbreak Attacks
  7. 2024BiasJailbreak:Analyzing Ethical Biases and Jailbreak Vulnerabilities in Large Language Models
  8. 2025Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check