mlsanityData to 7 Oct 2026

Visual Imitation Learning with Patch Rewards

arXiv 2302.00965 ↗First posted 2 Feb 2023cs.AIcited by 16

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Visual imitation learning enables reinforcement learning agents to learn to behave from expert visual demonstrations such as videos or image sequences, without explicit, well-defined rewards. Previous research either adopted supervised learning techniques or induce simple and coarse scalar rewards from pixels, neglecting the dense information contained in the image demonstrations. In this work, we propose to measure the expertise of various local regions of image samples, or called textitpatches, and recover multi-dimensional textitpatch rewards accordingly. Patch reward is a more precise rewarding characterization that serves as a fine-grained expertise measurement and visual explainability tool. Specifically, we present Adversarial Imitation Learning with Patch Rewards (PatchAIL), which employs a patch-based discriminator to measure the expertise of different local parts from given images and provide patch rewards. The patch-based knowledge is also used to regularize the aggregated reward and stabilize the training. We evaluate our method on DeepMind Control Suite and Atari tasks. The experiment results have demonstrated that PatchAIL outperforms baseline methods and provides valuable interpretations for visual demonstrations.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 48 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

33 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transport30 Jun
  2. 2022Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization4 Mar
  3. 2021Visual Adversarial Imitation Learning using Variational Models16 Jul
  4. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  5. 2020Energy-Based Imitation Learning20 Apr
  6. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  7. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  8. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2017Improved Training of Wasserstein GANscited by 2,223
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2018DeepMind Control Suitecited by 935
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
13 more
  1. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  2. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  3. 2020Reinforcement Learning with Augmented Datacited by 452
  4. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  5. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  6. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  7. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  8. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  9. 2016Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networkscited by 192
  10. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  11. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  12. 2018Zero-Shot Visual Imitationcited by 114
  13. 2022PerfectDou: Dominating DouDizhu with Perfect Information Distillationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 4; 2026: 1.

16 papers.

Sort:

By area:

  1. 2023Learning from Visual Observation via Offline Pretrained State-to-Go Transformercited by 14
  2. 2023SeMAIL: Eliminating Distractors in Visual Imitation via Separated Modelscited by 10
  3. 2023Adversarial Imitation Learning from Visual Observations using Latent Informationcited by 10
  4. 2024Visually Robust Adversarial Imitation Learning from Videos with Contrastive Learningcited by 9
  5. 2023Robust Visual Imitation Learning with Inverse Dynamics Representationscited by 8
  6. 2024Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videoscited by 4
  7. 2025Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representationscited by 4
  8. 2024NOLO: Navigate Only Look Oncecited by 3
  9. 2024Overcoming Knowledge Barriers: Online Imitation Learning from Visual Observation with Pretrained World Modelscited by 2
  10. 2024Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guaranteescited by 1
  11. 2025Beyond Domain Randomization: Event-Inspired Perception for Visually Robust Adversarial Imitation from Videoscited by 1
  12. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Explorationcited by 1
4 more
  1. 2026Dual-Granularity Contrastive Reward via Generated Episodic Guidance for Efficient Embodied RLcited by 1
  2. 2024Visual Imitation Learning with Calibrated Contrastive Representationcited by 0
  3. 2024A Model-Based Approach for Improving Reinforcement Learning Efficiency Leveraging Expert Observationscited by 0
  4. 2025Learning from Observation: A Survey of Recent Advancescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion-Reward Adversarial Imitation Learning
  2. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
  3. 2019Task-Relevant Adversarial Imitation Learning
  4. 2022Resolving Copycat Problems in Visual Imitation Learning via Residual Action Prediction
  5. 2018Injective State-Image Mapping facilitates Visual Adversarial Imitation Learning
  6. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation
  7. 2025GLUE: Global-Local Unified Encoding for Imitation Learning via Key-Patch Tracking
  8. 2025SD2AIL: Adversarial Imitation Learning from Synthetic Demonstrations via Diffusion Models