mlsanityData to 7 Oct 2026

PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards

arXiv 2007.15543 ↗First posted 30 Jul 2020cs.LG cs.AI stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL), particularly in sparse reward settings, often requires prohibitively large numbers of interactions with the environment, thereby limiting its applicability to complex problems. To address this, several prior approaches have used natural language to guide the agent's exploration. However, these approaches typically operate on structured representations of the environment, and/or assume some structure in the natural language commands. In this work, we propose a model that directly maps pixels to rewards, given a free-form natural language description of the task, which can then be used for policy learning. Our experiments on the Meta-World robot manipulation domain show that language-based rewards significantly improves the sample efficiency of policy learning, both in sparse and dense reward settings.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2020: 1; 2021: 4; 2022: 5; 2023: 9; 2024: 8; 2025: 3.

30 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  3. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  4. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  5. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  6. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  7. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  8. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  9. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  10. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 44
  11. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  12. 2020Learning Rewards from Linguistic Feedbackcited by 41
18 more
  1. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  2. 2024Natural Language Can Help Bridge the Sim2Real Gapcited by 21
  3. 2023Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Datacited by 20
  4. 2022Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Taskscited by 17
  5. 2021Zero-shot Task Adaptation using Natural Languagecited by 15
  6. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  7. 2022Learning from Symmetry: Meta-Reinforcement Learning with Symmetrical Behaviors and Language Instructionscited by 8
  8. 2024The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewardscited by 7
  9. 2024RLZero: Direct Policy Inference from Language Without In-Domain Supervisioncited by 7
  10. 2022RLang: A Declarative Language for Describing Partial World Knowledge to Reinforcement Learning Agentscited by 4
  11. 2023Language-guided Task Adaptation for Imitation Learningcited by 2
  12. 2023A Reminder of its Brittleness: Language Reward Shaping May Hinder Learning for Instruction Following Agentscited by 2
  13. 2023Improve the efficiency of deep reinforcement learning through semantic exploration guided by natural languagecited by 1
  14. 2025Mixed-Initiative Dialog for Human-Robot Collaborative Manipulationcited by 1
  15. 2025DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learningcited by 1
  16. 2024Language, Environment, and Robotic Navigationcited by 0
  17. 2024A Research Agenda for Usability and Generalisation in Reinforcement Learningcited by 0
  18. 2025NatSGLD: A Dataset with Speech, Gesture, Logic, and Demonstration for Robot Learning in Natural Human-Robot Interactioncited by 0