mlsanityData to 7 Oct 2026

Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learning

arXiv 2604.26516 ↗First posted 29 Apr 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Offline reinforcement learning (RL) agents often fail when deployed, as the gap between training datasets and real environments leads to unsafe behavior. To address this, we present SAS (Self-Alignment for Safety), a transformer-based framework that enables test-time adaptation in offline safe RL without retraining. In SAS, the main mechanism is self-alignment: at test time, the pretrained agent generates several imagined trajectories and selects those satisfying the Lyapunov condition. These feasible segments are then recycled as in-context prompts, allowing the agent to realign its behavior toward safety while avoiding parameter updates. In effect, SAS turns Lyapunov-guided imagination into control-invariant prompts, and its transformer architecture admits a hierarchical RL interpretation where prompting functions as Bayesian inference over latent skills. Across Safety Gymnasium and MuJoCo benchmarks, SAS consistently reduces cost and failure while maintaining or improving return.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 53 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

35 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model19 Jan
  2. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
  3. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning14 Feb
  4. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
19 more
  1. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  3. 2022Transformers are Sample-Efficient World Modelscited by 317
  4. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  6. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  7. 2020Neural Lyapunov Controlcited by 142
  8. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  9. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  10. 2020Conservative Safety Critics for Explorationcited by 107
  11. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  12. 2020DeepReach: A Deep Learning Approach to High-Dimensional Reachabilitycited by 81
  13. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  14. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  15. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  16. 2022Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Controlcited by 28
  17. 2021Density Constrained Reinforcement Learningcited by 25
  18. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  19. 2024SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learningcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning
  2. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  3. 2023Self-Confirming Transformer for Belief-Conditioned Adaptation in Offline Multi-Agent Reinforcement Learning
  4. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  5. 2026Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learning
  6. 2026SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment
  7. 2025RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory