GEM: Guided Expectation-Maximization for Behavior-Normalized Candidate Action Selection in Offline RL
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline reinforcement learning (RL) can fit strong value functions from fixed datasets, yet reliable deployment still hinges on the action selection interface used to query them. When the dataset induces a branched or multimodal action landscape, unimodal policy extraction can blur competing hypotheses and yield "in-between" actions that are weakly supported by data, making decisions brittle even with a strong critic. We introduce GEM (Guided Expectation-Maximization), an analytical framework that makes action selection both multimodal and explicitly controllable. GEM trains a Gaussian Mixture Model (GMM) actor via critic-guided, advantage-weighted EM-style updates that preserve distinct components while shifting probability mass toward high-value regions, and learns a tractable GMM behavior model to quantify support. During inference, GEM performs candidate-based selection: it generates a parallel candidate set and reranks actions using a conservative ensemble lower-confidence bound together with behavior-normalized support, where the behavior log-likelihood is standardized within each state's candidate set to yield stable, comparable control across states and candidate budgets. Empirically, GEM is competitive across D4RL benchmarks, and offers a simple inference-time budget knob (candidate count) that trades compute for decision quality without retraining.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 18 references that are in the map. First posted on arXiv 24 Mar 2026.
Builds on
17 of 18 references.
Closest ancestors · 10
Same direction, newest first
- 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL26 May
- 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
- 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
- 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020MOReL : Model-Based Offline Reinforcement Learning12 May
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2019Behavior Regularized Offline Reinforcement Learning26 Nov
Other references · 7
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
- 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
- 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
- 2026ReBRAC-v2: The Return of the King
- 2026Decoupling Policy Extraction for Offline Reinforcement Learning
- 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
- 2023The In-Sample Softmax for Offline Reinforcement Learning
- 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
- 2026Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learning
- 2025GEM: A Gym for Agentic LLMs