mlsanityData to 14 Oct 2026

Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness

arXiv 2608.09900 ↗First posted 10 Aug 2026cs.CLcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Large language model evaluations typically focus on performance under nominal conditions, creating an illusion of capability where models comfortably walk a narrow, highly optimized generation corridor. In real-world deployments, however, complex system prompts, safety guardrails, and structural constraints continuously force models off this nominal path, driving a divergence between benchmark scores and deployment performance. To address this issue, we introduce Decoding-Level Taboo, a zero-prompt diagnostic stress test that intervenes directly in logit space at runtime, forcing models out of their nominal paths. By dynamically masking primary candidate tokens at word boundaries, Taboo forces machine circumlocution. Evaluating Taboo across several open-weight model families reveals that off-path robustness is heavily influenced by both parameter scale and post-training instruction alignment, with robustness generally improving with model size and alignment. Beyond the results presented in this paper, Taboo provides a novel primitive for generating diverse synthetic datasets, stress-testing runtime safety guardrails, and auditing model reliability prior to real-world deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 13 of its 20 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

13 of 20 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  12. 2026Say Anything but This: When Tokenizer Betrays Reasoning in LLMscited by 1
1 more
  1. 2026Trivial Vocabulary Bans Improve LLM Reasoning More Than Deep Linguistic Constraintscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating LLM Safety Under Repeated Inference via Accelerated Prompt Stress Testing
  2. 2026Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations
  3. 2025Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models
  4. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMs
  5. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  6. 2026Detecting Safety Training Modification in Language Models via Activation Analysis ↗
  7. 2026LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling
  8. 2025LLM-Safety Evaluations Lack Robustness