mlsanityData to 7 Oct 2026

Should LLM Safety Be More Than Refusing Harmful Instructions?

arXiv 2506.02442 ↗First posted 3 Jun 2025cs.CLcited by 6

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

This paper presents a systematic evaluation of Large Language Models' (LLMs) behavior on long-tail distributed (encrypted) texts and their safety implications. We introduce a two-dimensional framework for assessing LLM safety: (1) instruction refusal-the ability to reject harmful obfuscated instructions, and (2) generation safety-the suppression of generating harmful responses. Through comprehensive experiments, we demonstrate that models that possess capabilities to decrypt ciphers may be susceptible to mismatched-generalization attacks: their safety mechanisms fail on at least one safety dimension, leading to unsafe responses or over-refusal. Based on these findings, we evaluate a number of pre-LLM and post-LLM safeguards and discuss their strengths and limitations. This work contributes to understanding the safety of LLM in long-tail text scenarios and provides directions for developing robust safety mechanisms.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 35 of its 58 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

35 of 58 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Endless Jailbreaks with Bijection Learning2 Oct
  2. 2024Jailbreaking Large Language Models with Symbolic Mathematics17 Sep
  3. 2024Efficient Adversarial Training in LLMs with Continuous Attacks24 May
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  5. 2024When "Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Ciphers16 Feb
  6. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  7. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  8. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  9. 2023Detecting Language Model Attacks with Perplexity27 Aug
  10. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  11. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2023Mistral 7Bcited by 3,065
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
12 more
  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  3. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  4. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  5. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  6. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  7. 2023TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Taskscited by 48
  8. 2022SALSA: Attacking Lattice Cryptography with Transformerscited by 28
  9. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methodscited by 27
  10. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  11. 2025CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challengescited by 11
  12. 2025Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilitiescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Too Helpful, Too Harmless, Too Honest or Just Right?cited by 9
  2. 2025DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Modelscited by 5
  3. 2026Do Large Language Models Reflect Demographic Pluralism in Safety?cited by 1
  4. 2026When the Model Said 'No Comment', We Knew Helpfulness Was Dead, Honesty Was Alive, and Safety Was Terrifiedcited by 1
  5. 2026Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMscited by 1
  6. 2025InvThink: Premortem Reasoning for Safer Language Modelscited by 0