Agent-Aware Training for Agent-Agnostic Action Advising in Deep Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Action advising endeavors to leverage supplementary guidance from expert teachers to alleviate the issue of sampling inefficiency in Deep Reinforcement Learning (DRL). Previous agent-specific action advising methods are hindered by imperfections in the agent itself, while agent-agnostic approaches exhibit limited adaptability to the learning agent. In this study, we propose a novel framework called Agent-Aware trAining yet Agent-Agnostic Action Advising (A7) to strike a balance between the two. The underlying concept of A7 revolves around utilizing the similarity of state features as an indicator for soliciting advice. However, unlike prior methodologies, the measurement of state feature similarity is performed by neither the error-prone learning agent nor the agent-agnostic advisor. Instead, we employ a proxy model to extract state features that are both discriminative (adaptive to the agent) and generally applicable (robust to agent noise). Furthermore, we utilize behavior cloning to train a model for reusing advice and introduce an intrinsic reward for the advised samples to incentivize the utilization of expert guidance. Experiments are conducted on the GridWorld, LunarLander, and six prominent scenarios from Atari games. The results demonstrate that A7 significantly accelerates the learning process and surpasses existing methods (both agent-specific and agent-agnostic) by a substantial margin. Our code will be made publicly available.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 28 Nov 2023.
Builds on
35 of 45 references.
References · 35
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2023GPT-4 Technical Reportcited by 18,132
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2019Contrastive Multiview Codingcited by 1,297
23 more
- 2018Exploration by Random Network Distillationcited by 966
- 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
- 2018Learning Dexterous In-Hand Manipulationcited by 643
- 2016Deep Exploration via Bootstrapped DQNcited by 518
- 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
- 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
- 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
- 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 231
- 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
- 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
- 2019Model-free Deep Reinforcement Learning for Urban Autonomous Drivingcited by 96
- 2020Inverse Reinforcement Learning with Natural Language Goalscited by 20
- 2020A Conceptual Framework for Externally-influenced Agents: An Assisted Reinforcement Learning Reviewcited by 15
- 2019Teaching on a Budget in Multi-Agent Deep Reinforcement Learningcited by 14
- 2021Action Advising with Advice Imitation in Deep Reinforcement Learningcited by 10
- 2020Student-Initiated Action Advising via Advice Noveltycited by 7
- 2019Deep Q-Learning with Q-Matrix Transfer Learning for Novel Fire Evacuation Environmentcited by 6
- 2020Smart Train Operation Algorithms based on Expert Knowledge and Reinforcement Learningcited by 5
- 2021Persistent Rule-based Interactive Reinforcement Learningcited by 5
- 2022Methodical Advice Collection and Reuse in Deep Reinforcement Learningcited by 1
- 2022Ask-AC: An Initiative Advisor-in-the-Loop Actor-Critic Frameworkcited by 1
- 2022Quantifying the Effect of Feedback Frequency in Interactive Reinforcement Learning for Robotic Taskscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.