mlsanityData to 14 Oct 2026

CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning

arXiv 2608.07719 ↗First posted 7 Aug 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning repeatedly trains policies from a fixed transition pool, making redundant data costly across seeds and hyperparameters, while naive subsampling can remove rare transitions needed for long-horizon credit assignment. We introduce CODS, a critic-guided selector that alternates between fitting an algorithm-matched critic and acquiring high-residual transitions before freezing a reusable subset. Unlike prioritized replay, CODS produces a static artifact; unlike one-shot residual selection, it refreshes scores as the critic changes. At a 10% budget, CODS retains 96.6% of eligible-pool performance across 20 valid D4RL task--algorithm cells. It exceeds ReDOR and OPER on 19/20 cells and every other subset baseline on 20/20; all six subset advantages remain significant under predeclared hierarchical inference with Holm correction. Holding total selector updates fixed, five acquisition rounds improve four representative cells by 11.23 points over one round and saturate thereafter. Equal-pass and equal-hour evaluations clarify that reuse, rather than a single-run speedup, creates the compute advantage. Mechanism and corruption interventions expose both useful sparse-reward enrichment and sensitivity to outliers. Finally, a whole-trace extension retains 95.4% of pooled ALFWorld success and 96.5% of pooled GSM8K exact match. CODS is therefore a reusable selection procedure, not a formal coreset guarantee.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 23 references that are in the map. First posted on arXiv 7 Aug 2026.

Builds on

14 of 23 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset26 Feb
  2. 2022Boosting Offline Reinforcement Learning via Data Rebalancing17 Oct
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 8

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  3. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  6. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  7. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  8. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Residuals-based Offline Reinforcement Learning
  2. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  3. 2025A Clean Slate for Offline Reinforcement Learning
  4. 2026Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
  5. 2026WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
  6. 2020Critic Regularized Regression
  7. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
  8. 2021Representation Matters: Offline Pretraining for Sequential Decision Making