mlsanityData to 14 Oct 2026

A Bird's-Eye View of Iterative Reward Design

arXiv 2610.04364 ↗First posted 3 Oct 2026cs.LG cs.AI cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Designing effective reward functions in RL typically requires substantial expertise and trial and error. Recent work automates this process with LLM-based systems that generate and iteratively improve reward code using policy feedback. However, these methods are often hard to compare because they differ in implementation details, feedback assumptions, and evaluation environments. To address this, we introduce a Benchmark for Iterative Reward Design (BIRD) that expresses existing methods in a unified configuration and evaluation space. This lets us compare algorithms directly, ablate individual design choices, and prototype new components under matched feedback conditions and policy-training budgets. Across MuJoCo, Meta-World, Assistax, and HumanoidBench, we identify a small set of simple design choices that consistently improve performance. Combining these choices yields significantly better performance than the evaluated methods from prior work. Our results highlight the strength of simple baselines and motivate further study of when additional algorithmic complexity improves iterative reward design. Code is available at https://github.com/safety-research/bird.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 52 references that are in the map. First posted on arXiv 3 Oct 2026.

Builds on

28 of 52 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  6. 2015Illuminating search spaces by mapping elitescited by 425
  7. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  8. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  9. 2022Defining and Characterizing Reward Hackingcited by 203
  10. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 160
  11. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  12. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
16 more
  1. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 83
  2. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  3. 2020Quantifying Differences in Reward Functionscited by 63
  4. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 61
  5. 2022Predictive Sampling: Real-time Behaviour Synthesis with MuJoCocited by 47
  6. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 44
  7. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 38
  8. 2019Evolving Rewards to Automate Reinforcement Learningcited by 31
  9. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 30
  10. 2024REvolve: Reward Evolution with Large Language Models using Human Feedbackcited by 23
  11. 2025A Clean Slate for Offline Reinforcement Learningcited by 11
  12. 2024Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolutioncited by 5
  13. 2026RF-Agent: Automated Reward Function Design via Language Agent Tree Searchcited by 4
  14. 2025Automated Reward Design for Gran Turismocited by 3
  15. 2026RDA: Reward Design Agent for Reinforcement Learningcited by 3
  16. 2026Discovering Reinforcement Learning Interfaces with Large Language Modelscited by 2

Built on by

Not cited yet.