mlsanityData to 7 Oct 2026

Less is More: Clustered Cross-Covariance Control for Offline RL

arXiv 2601.20765 ↗First posted 28 Jan 2026cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

A fundamental challenge in offline reinforcement learning is distributional shift. Scarce data or datasets dominated by out-of-distribution (OOD) areas exacerbate this issue. Our theoretical analysis and experiments show that the standard squared error objective induces a harmful TD cross covariance. This effect amplifies in OOD areas, biasing optimization and degrading policy learning. To counteract this mechanism, we develop two complementary strategies: partitioned buffer sampling that restricts updates to localized replay partitions, attenuates irregular covariance effects, and aligns update directions, yielding a scheme that is easy to integrate with existing implementations, namely Clustered Cross-Covariance Control for TD (C^4). We also introduce an explicit gradient-based corrective penalty that cancels the covariance induced bias within each update. We prove that buffer partitioning preserves the lower bound property of the maximization objective, and that these constraints mitigate excessive conservatism in extreme OOD areas without altering the core behavior of policy constrained offline reinforcement learning. Empirically, our method showcases higher stability and up to 30% improvement in returns over prior methods, especially with small datasets and splits that emphasize OOD areas.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 67 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

22 of 67 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data2 Dec
  2. 2025Policy-Based Trajectory Clustering in Offline Reinforcement Learning10 Jun
  3. 2024Dataset Clustering for Improved Offline Policy Learning14 Feb
  4. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators29 Nov
  5. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size20 Nov
  6. 2022Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficient3 Oct
  7. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation21 Nov
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 12

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  4. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  5. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  6. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  7. 2018Variational Option Discovery Algorithmscited by 121
  8. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  9. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  10. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  11. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  12. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026T5: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  2. 2022Offline Policy Optimization in RL with Variance Regularizaton
  3. 2024CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning
  4. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithm
  5. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
  6. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL
  7. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
  8. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond