mlsanityData to 14 Oct 2026

Learning Safety Constraints for Large Language Models

arXiv 2505.24445 ↗First posted 30 May 2025cs.LG cs.AIcited by 13

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Large language models (LLMs) have emerged as powerful tools but pose significant safety risks through harmful outputs and vulnerability to adversarial attacks. We propose SaP, short for Safety Polytope, a geometric approach to LLM safety that learns and enforces multiple safety constraints directly in the model's representation space. We develop a framework that identifies safe and unsafe regions via the polytope's facets, enabling both detection and correction of unsafe outputs through geometric steering. Unlike existing approaches that modify model weights, SaP operates post-hoc in the representation space, preserving model capabilities while enforcing safety constraints. Experiments across multiple LLMs demonstrate that our method can effectively detect unethical inputs, reduce adversarial attack success rates while maintaining performance on standard tasks, thus highlighting the importance of having an explicit geometric model for safety. Analysis of the learned polytope facets reveals emergence of specialization in detecting different semantic notions of safety, providing interpretable insights into how safety is captured in LLMs' representation space.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 53 of its 62 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

53 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
  2. 2024Improving Alignment and Robustness with Circuit Breakers6 Jun

Other references · 51

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  8. 2022Toy Models of Superpositioncited by 971
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  10. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
38 more
  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  5. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  7. 2019Differentiable Convex Optimization Layerscited by 465
  8. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  9. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  10. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  11. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  12. 2018Safe Exploration in Continuous Action Spacescited by 247
  13. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  14. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  15. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  16. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  17. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  18. 2021Efficient and Modular Implicit Differentiationcited by 203
  19. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 192
  20. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  21. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  22. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  23. 2024Token-level Direct Preference Optimizationcited by 135
  24. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
  25. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 96
  26. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  27. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  28. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  29. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 70
  30. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 49
  31. 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetitioncited by 35
  32. 2022Interpreting Neural Networks through the Polytope Lenscited by 33
  33. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  34. 2023Learning Safety Constraints from Demonstrations with Unknown Rewardscited by 19
  35. 2023SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtrackingcited by 13
  36. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9
  37. 2024MPAX: Mathematical Programming in JAXcited by 7
  38. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  2. 2025Preemptive Detection and Steering of LLM Misalignment via Latent Reachabilitycited by 10
  3. 2025Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depthcited by 3
  4. 2025Countermind: A Multi-Layered Security Architecture for Large Language Modelscited by 2
  5. 2026MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Followingcited by 2
  6. 2025From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrailscited by 1
  7. 2025Optimal Anytime Algorithms for Online Convex Optimization with Adversarial Constraintscited by 1
  8. 2025ProSocialAlign: Preference Conditioned Test Time Alignment in Language Modelscited by 1
  9. 2026Beyond Labels: Information-Efficient Human-in-the-Loop Learning using Ranking and Selection Queriescited by 1
  10. 2026Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoderscited by 1
  11. 2025Foundation Models for Logistics: Toward Certifiable, Conversational Planning Interfacescited by 0
  12. 2026BarrierSteer: LLM Safety via Learning Barrier Steeringcited by 0
1 more
  1. 2026Local Sparsity Enables Unsupervised LLM Safety Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  2. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signals
  3. 2025Advancing LLM Safe Alignment with Safety Representation Ranking
  4. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  5. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  6. 2026Safety Targeted Embedding Exploit via Refinement
  7. 2026Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety
  8. 2026Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks