mlsanityData to 14 Oct 2026

Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning

arXiv 2506.11172 ↗First posted 12 Jun 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) heavily relies on the coverage of pre-collected data over the target policy's distribution. Existing studies aim to improve data-policy coverage to mitigate distributional shifts, but overlook security risks from insufficient coverage, and the single-step analysis is not consistent with the multi-step decision-making nature of offline RL. To address this, we introduce the sequence-level concentrability coefficient to quantify coverage, and reveal its exponential amplification on the upper bound of estimation errors through theoretical analysis. Building on this, we propose the Collapsing Sequence-Level Data-Policy Coverage (CSDPC) poisoning attack. Considering the continuous nature of offline RL data, we convert state-action pairs into decision units, and extract representative decision patterns that capture multi-step behavior. We identify rare patterns likely to cause insufficient coverage, and poison them to reduce coverage and exacerbate distributional shifts. Experiments show that poisoning just 1% of the dataset can degrade agent performance by 90%. This finding provides new perspectives for analyzing and safeguarding the security of offline RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 43 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

31 of 43 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets7 Oct
  2. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  9. 2019Policy Poisoning in Batch Reinforcement Learning and Control13 Oct

Other references · 22

  1. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  8. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  9. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  10. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  11. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  12. 2017Delving into adversarial attacks on deep policiescited by 104
10 more
  1. 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
  2. 2020Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learningcited by 63
  3. 2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 55
  4. 2020Backpropagated Gradient Representations for Anomaly Detectioncited by 54
  5. 2020Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamicscited by 38
  6. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26
  7. 2022Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learningcited by 21
  8. 2023Reinforcement Learning with Simple Sequence Priorscited by 11
  9. 2019Minimalistic Attacks: How Little it Takes to Fool a Deep Reinforcement Learning Policycited by 10
  10. 2023SoK: Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimal Perturbation Budget Allocation for Data Poisoning in Offline Reinforcement Learning
  2. 2025Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning
  3. 2026Safe-RULE: Safe Reinforcement UnLEarning
  4. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  5. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  6. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling
  7. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  8. 2026UBA-ORL: Unlearning-Activated Backdoor Attacks on Offline Reinforcement Learning ↗