mlsanityData to 7 Oct 2026

RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization

arXiv 2510.02695 ↗First posted 3 Oct 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In safety-critical domains where online data collection is infeasible, offline reinforcement learning (RL) is attractive only if policies achieve high returns without catastrophic lower-tail risk. Prior work on risk-averse offline RL achieves safety at the cost of either (i) value/model-based pessimism or (ii) restricted policy classes that limit expressiveness, whereas diffusion/flow-based expressive generative policies have largely been used in risk-neutral settings. We introduce Risk-Aware Multimodal Actor-Critic (RAMAC), a simple, modular, model-free framework that couples an expressive generative actor (e.g., diffusion/flow) with a distributional critic and optimizes a composite objective that combines Conditional Value-at-Risk (CVaR) with behavioral cloning (BC), enabling risk-sensitive learning in complex multimodal scenarios. Since out-of-distribution (OOD) actions are a major driver of catastrophic failures in offline RL, we further provide an objective-level analysis showing that controlling behavior divergence via BC suppresses OOD actions and stabilizes CVaR. Instantiating RAMAC with a diffusion actor, we illustrate these insights on a 2-D risky bandit and evaluate on Stochastic-D4RL, observing consistent gains in CVaR0.1 while maintaining strong returns. The code and experimental results are available on the hrefhttps://kaifukazawa.github.io/ramac-project/ project website

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 57 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

50 of 57 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning26 Mar
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  3. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning30 Nov
  4. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  6. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020MOPO: Model-based Offline Policy Optimization27 May
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
2 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 36

  1. 2020Denoising Diffusion Implicit Modelscited by 9,655
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2022Flow Matching for Generative Modelingcited by 5,426
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
24 more
  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  4. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  5. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  6. 2025Flow Q-Learningcited by 162
  7. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  8. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  9. 2024FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulationcited by 99
  10. 2014Optimizing the CVaR via Samplingcited by 98
  11. 2015Policy Gradient for Coherent Risk Measurescited by 74
  12. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  13. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
  14. 2025ManiFlow: A General Robot Manipulation Policy via Consistency Flow Trainingcited by 43
  15. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learningcited by 41
  16. 2024JUICER: Data-Efficient Imitation Learning for Robotic Assemblycited by 36
  17. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  18. 2025VITA: Vision-to-Action Flow Matching Policycited by 23
  19. 2024InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulationcited by 21
  20. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  21. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learningcited by 12
  22. 2025Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformerscited by 8
  23. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimizationcited by 7
  24. 2025Distributional Soft Actor-Critic with Diffusion Policycited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
  2. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  3. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  4. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  5. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  6. 2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning
  7. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  8. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning