A Reminder of its Brittleness: Language Reward Shaping May Hinder Learning for Instruction Following Agents
Not on the map. Why?
Abstract (arXiv)
Teaching agents to follow complex written instructions has been an important yet elusive goal. One technique for enhancing learning efficiency is language reward shaping (LRS). Within a reinforcement learning (RL) framework, LRS involves training a reward function that rewards behaviours precisely aligned with given language instructions. We argue that the apparent success of LRS is brittle, and prior positive findings can be attributed to weak RL baselines. Specifically, we identified suboptimal LRS designs that reward partially matched trajectories, and we characterised a novel reward perturbation to capture this issue using the concept of loosening task constraints. We provided theoretical and empirical evidence that agents trained using LRS rewards converge more slowly compared to pure RL agents. Our work highlights the brittleness of existing LRS methods, which has been overlooked in the previous studies.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 50 of its 69 references that are in the map. First posted on arXiv 26 May 2023.
Builds on
50 of 69 references.
References · 50
- 2017Attention Is All You Needcited by 59,932
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
- 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
- 2020Mastering Atari with Discrete World Modelscited by 990
- 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
38 more
- 2018Exploration by Random Network Distillationcited by 966
- 2017Adversarial Attacks on Neural Network Policiescited by 314
- 2018Reward learning from human preferences and demonstrations in Ataricited by 314
- 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
- 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
- 2022Masked World Models for Visual Controlcited by 169
- 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
- 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
- 2022On the Convergence Rates of Policy Gradient Methodscited by 115
- 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
- 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
- 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
- 2020Hard negative examples are hard, but usefulcited by 82
- 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
- 2021On the Expressivity of Markov Rewardcited by 71
- 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
- 2021BACKDOORL: Backdoor Attack against Competitive Reinforcement Learningcited by 66
- 2022Multi-modal Alignment using Representation Codebookcited by 63
- 2020Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learningcited by 61
- 2020Attention over learned object embeddings enables complex visual reasoningcited by 59
- 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
- 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 54
- 2020Grounded Language Learning Fast and Slowcited by 54
- 2022Inferring Rewards from Language in Contextcited by 50
- 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
- 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
- 2017Beating Atari with Natural Language Guided Reinforcement Learningcited by 24
- 2020Safe Reinforcement Learning with Natural Language Constraintscited by 22
- 2017Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularitiescited by 19
- 2022Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Taskscited by 17
- 2022DANLI: Deliberative Agent for Following Natural Language Instructionscited by 15
- 2021Subgoal-based Reward Shaping to Improve Efficiency in Reinforcement Learningcited by 10
- 2021Expressing and Exploiting the Common Subgoal Structure of Classical Planning Domains Using Sketches: Extended Versioncited by 9
- 2022Efficient Reward Poisoning Attacks on Online Deep Reinforcement Learningcited by 9
- 2019A new Potential-Based Reward Shaping for Reinforcement Learning Agentcited by 8
- 2022Compositionality as Lexical Symmetrycited by 8
- 2019Pre-Learning Environment Representations for Data-Efficient Neural Instruction Followingcited by 6
- 2022Reward Delay Attacks on Deep Reinforcement Learningcited by 5
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.