# Automated Rewards via LLM-Generated Progress Functions

arXiv: https://arxiv.org/abs/2410.09187 · HTML: https://mlsanity.com/paper/2410.09187/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [RoboCLIP: One Demonstration is Enough to Learn Robot Policies](https://mlsanity.com/paper/2310.07899/) (2023)
- [Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning](https://mlsanity.com/paper/2206.08686/) (2022)
- [OMNI: Open-endedness via Models of human Notions of Interestingness](https://mlsanity.com/paper/2306.01711/) (2023)
- [Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft](https://mlsanity.com/paper/2312.09238/) (2023)
- [Positional Description Matters for Transformers Arithmetic](https://mlsanity.com/paper/2311.14737/) (2023)
- [Dealing with Sparse Rewards in Reinforcement Learning](https://mlsanity.com/paper/1910.09281/) (2019)
- [ELLA: Exploration through Learned Language Abstraction](https://mlsanity.com/paper/2103.05825/) (2021)
- [Twisting Lids Off with Two Hands](https://mlsanity.com/paper/2403.02338/) (2024)
- [Code as Reward: Empowering Reinforcement Learning with VLMs](https://mlsanity.com/paper/2402.04764/) (2024)

## Built on by (most cited first, 4 of 4)

- [Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning](https://mlsanity.com/paper/2504.00907/) (2025)
- [Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Exploration](https://mlsanity.com/paper/2601.10744/) (2026)
- [Leveraging LLMs for reward function design in reinforcement learning control tasks](https://mlsanity.com/paper/2511.19355/) (2025)
- [Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions](https://mlsanity.com/paper/2509.07445/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have the potential to automate reward engineering by leveraging their broad domain knowledge across various tasks. However, they often need many iterations of trial-and-error to generate effective reward functions. This process is costly because evaluating every sampled reward function requires completing the full policy optimization process for each function. In this paper, we introduce an LLM-driven reward generation framework that is able to produce state-of-the-art policies on the challenging Bi-DexHands benchmark with 20x fewer reward function samples than the prior state-of-the-art work. Our key insight is that we reduce the problem of generating task-specific rewards to the problem of coarsely estimating task progress. Our two-step solution leverages the task domain knowledge and the code synthesis abilities of LLMs to author progress functions that estimate task progress from a given state. Then, we use this notion of progress to discretize states, and generate count-based intrinsic rewards using the low-dimensional state space. We show that the combination of LLM-generated progress functions and count-based intrinsic rewards is essential for our performance gains, while alternatives such as generic hash-based counts or using progress directly as a reward function fall short.