mlsanityData to 7 Oct 2026

Learning to Identify Critical States for Reinforcement Learning from Videos

arXiv 2308.07795 ↗First posted 15 Aug 2023cs.CV cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Recent work on deep reinforcement learning (DRL) has pointed out that algorithmic information about good policies can be extracted from offline data which lack explicit information about executed actions. For example, videos of humans or robots may convey a lot of implicit information about rewarding action sequences, but a DRL machine that wants to profit from watching such videos must first learn by itself to identify and recognize relevant states/actions/rewards. Without relying on ground-truth annotations, our new method called Deep State Identifier learns to predict returns from episodes encoded as videos. Then it uses a kind of mask-based sensitivity analysis to extract/identify important critical states. Extensive experiments showcase our method's potential for understanding and improving agent behavior. The source code and the generated datasets are available at https://github.com/AI-Initiative-KAUST/VideoRLCS.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 52 of its 83 references that are in the map. First posted on arXiv 15 Aug 2023.

Builds on

52 of 83 references.

References · 52

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2018World Modelscited by 1,444
  11. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
38 more
  1. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  2. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  3. 2021Domain Generalization: A Surveycited by 760
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  5. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  6. 2016Rationalizing Neural Predictionscited by 281
  7. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  8. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  9. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  10. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  11. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 248
  12. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  13. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  14. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  15. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  16. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  17. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  18. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  19. 2021Salient Object Detection via Integrity Learningcited by 133
  20. 2017Emergent Complexity via Multi-Agent Competitioncited by 117
  21. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  22. 2020Video Self-Stitching Graph Network for Temporal Action Localizationcited by 104
  23. 2017Weakly Supervised Action Localization by Sparse Temporal Pooling Networkcited by 100
  24. 2018Self-Erasing Network for Integral Object Attentioncited by 95
  25. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  26. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  27. 2022Cross Language Image Matching for Weakly Supervised Semantic Segmentationcited by 72
  28. 2015On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Modelscited by 52
  29. 2020Neuroevolution of Self-Interpretable Agentscited by 48
  30. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  31. 2020Policy Evaluation Networkscited by 28
  32. 2020Parameter-Based Value Functionscited by 24
  33. 2022Annotated History of Modern AI and Deep Learningcited by 20
  34. 2022Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localizationcited by 19
  35. 2018Explaining Deep Learning Models - A Bayesian Non-parametric Approachcited by 16
  36. 2018One Big Net For Everythingcited by 13
  37. 2023Guiding Online Reinforcement Learning with Action-Free Offline Pretrainingcited by 11
  38. 2020DANCE: Enhancing saliency maps using decoyscited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 5.

9 papers.

  1. 2024Faster Diffusion via Temporal Attention Decompositioncited by 54
  2. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  3. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
  4. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
  5. 2024SkiLD: Unsupervised Skill Discovery Guided by Factor Interactionscited by 9
  6. 2025Behaviour Discovery and Attribution for Explainable Reinforcement Learningcited by 2
  7. 2025Exploiting Policy Idling for Dexterous Manipulationcited by 1
  8. 2025From Actions to Words: Towards Abstractive-Textual Policy Summarization in RLcited by 0
  9. 2025Active Query Selection for Crowd-Based Reinforcement Learningcited by 0