mlsanityData to 7 Oct 2026

GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning

arXiv 2510.02180 ↗First posted 2 Oct 2025cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Inverse Reinforcement Learning aims to recover reward models from expert demonstrations, but traditional methods yield black-box models that are difficult to interpret and debug. In this work, we introduce GRACE (Generating Rewards As CodE), a method for using Large Language Models within an evolutionary search to reverse-engineer an interpretable, code-based reward function directly from expert trajectories. The resulting reward function is executable code that can be inspected and verified. We empirically validate GRACE on the MuJoCo, BabyAI and AndroidWorld benchmarks, where it efficiently learns highly accurate rewards, even in complex, multi-task settings. Further, we demonstrate that the resulting reward leads to strong policies, compared to both competitive Imitation Learning and online RL approaches with ground-truth rewards. Finally, we show that GRACE is able to build complex reward APIs in multi-task setups.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 57 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

31 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Critique-out-Loud Reward Models21 Aug
  2. 2023Eureka: Human-Level Reward Design via Coding Large Language Models19 Oct
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  9. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  10. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  11. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  12. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
16 more
  1. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  2. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  3. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  4. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  5. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  6. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  7. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  8. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  9. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  10. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  11. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  12. 2024OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Codecited by 67
  13. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
  14. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 38
  15. 2024Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Searchcited by 38
  16. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
  2. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
  3. 2021Versatile Inverse Reinforcement Learning via Cumulative Rewards
  4. 2024Imitating Language via Scalable Inverse Reinforcement Learning
  5. 2024REvolve: Reward Evolution with Large Language Models using Human Feedback
  6. 2026Inverse RL Helps Align AI by Imitating Humans
  7. 2026CORE: Code-based Inverse Self-Training Framework with Graph Expansion for Virtual Agents
  8. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning