Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making
Not on the map. Why?
Abstract (arXiv)
Target-conditioned sequence models provide a simple interface for controllable offline decision making, but the requested target return can be an unreliable control signal, especially when the target return lies in underrepresented regions of the dataset. This paper proposes Doctor, a hybrid sequence modeling and reinforced verification framework for controllable target-conditioned offline decision making. Doctor trains a shared masked trajectory Transformer with two complementary objectives: masked trajectory reconstruction for candidate generation and in-sample value learning for action-value verification. At inference time, the model samples multiple nearby target returns, generates candidate actions in parallel, and selects the action whose verified value is closest to the requested target return. We analyze this verifier-guided selection rule and show that its value-level alignment error is bounded by candidate-value coverage around the target return and verifier accuracy. Experiments on D4RL and EpiCare show that Doctor improves target-return alignment under reduced high-return coverage, remains competitive on standard offline return-maximization benchmarks, and enables a single policy to modulate between conservative and aggressive operating points in a simulated clinical decision-making task. These results suggest that reinforced verification can improve the controllability of target-conditioned policies.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 30 references that are in the map. First posted on arXiv 22 Aug 2025.
Builds on
16 of 30 references.
References · 16
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
- 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
- 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
- 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
- 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
- 2024Theoretical guarantees on the best-of-n alignment policycited by 116
- 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
- 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
4 more
- 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
- 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
- 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
- 2024Enhancing textual textbook question answering with large language models and retrieval augmented generationcited by 13
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Reinformer: Max-Return Sequence Modeling for Offline RL
- 2025Target-Aligned Fusion for Decision-Sequence Learning under Dynamics Shift
- 2023Goal-Conditioned Predictive Coding for Offline Reinforcement Learning
- 2023Critic-Guided Decision Transformer for Offline Reinforcement Learning
- 2024Q-value Regularized Transformer for Offline Reinforcement Learning
- 2024M3PC: Test-time Model Predictive Control for Pretrained Masked Trajectory Model
- 2024Goal-Conditioned Decision Transformer for Multi-Goal Offline Reinforcement Learning
- 2022Towards Flexible Inference in Sequential Decision Problems via Bidirectional Transformers