mlsanityData to 7 Oct 2026

On Almost Surely Safe Alignment of Large Language Models at Inference-Time

arXiv 2502.01208 ↗First posted 3 Feb 2025cs.LG cs.CLcited by 13

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We introduce a novel inference-time alignment approach for LLMs that aims to generate safe responses almost surely, i.e., with probability approaching one. Our approach models the generation of safe responses as a constrained Markov Decision Process (MDP) within the LLM's latent space. We augment a safety state that tracks the evolution of safety constraints and dynamically penalize unsafe generations to ensure the generation of safe responses. Consequently, we demonstrate formal safety guarantees w.r.t. the given cost model upon solving the MDP in the latent space with sufficiently large penalties. Building on this foundation, we propose InferenceGuard, a practical implementation that safely aligns LLMs without modifying the model weights. Empirically, we demonstrate that InferenceGuard effectively balances safety and task performance, outperforming existing inference-time alignment methods in generating safe and aligned responses. Our findings contribute to the advancement of safer LLM deployment through alignment at inference-time, thus presenting a promising alternative to resource-intensive, overfitting-prone alignment techniques like RLHF.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 92 of its 119 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

92 of 119 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction9 Jan
  2. 2025Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information2 Jan
  3. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  4. 2024Aligning Large Language Models with Representation Editing: A Control Perspective10 Jun
  5. 2024Value Augmented Sampling for Language Model Alignment and Personalization10 May
  6. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts12 Feb
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  8. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  9. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  10. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  12. 2023Preference Ranking Optimization for Human Alignment30 Jun
1 more
  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 79

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
38 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 957
  2. 2017Constrained Policy Optimizationcited by 928
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  6. 2022PAL: Program-aided Language Modelscited by 635
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  9. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  11. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  12. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  13. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  14. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  15. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
  16. 2018Safe Exploration in Continuous Action Spacescited by 237
  17. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  18. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  19. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  20. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  21. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  22. 2023Controlled Decoding from Language Modelscited by 133
  23. 2023Large Language Models for Robotics: A Surveycited by 132
  24. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  25. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  26. 2018Learning-based Model Predictive Control for Safe Explorationcited by 116
  27. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  28. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  29. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  30. 2020Conservative Safety Critics for Explorationcited by 107
  31. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  32. 2022Simulation-guided Beam Search for Neural Combinatorial Optimizationcited by 84
  33. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  34. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  35. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  36. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72
  37. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  38. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  2. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuningcited by 2
  3. 2026Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplerscited by 2
  4. 2025Personalized Constitutionally-Aligned Agentic Superego: Secure AI Behavior Aligned to Diverse Human Valuescited by 1
  5. 2025Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspectivecited by 1
  6. 2025ExpertGenQA: Open-ended QA generation in Specialized Domainscited by 0
  7. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  8. 2025Safety Game: Inference-Time Alignment of Black-Box LLMs via Constrained Optimizationcited by 0
  9. 2026Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routingcited by 0
  10. 2026Safety Hacking in Constrained Best-of-N Inference-time Scalingcited by 0
  11. 2026Beam Search as Test-Time Self-Distillation via Counterfactual Contextscited by 0
  12. 2026Does This Action Still Explain the Task? Reverse Scoring for Diffusion Language Model Agentscited by 0
1 more
  1. 2026The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DNAlign: Dynamic Null-Space Safe Alignment for LLMs
  2. 2025Safety Alignment Depth in Large Language Models: A Markov Chain Perspective
  3. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  4. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
  5. 2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment
  6. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
  7. 2026Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Time
  8. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching