# Self-correcting Reward Shaping via Language Models for Reinforcement Learning Agents in Games

arXiv: https://arxiv.org/abs/2506.23626 · HTML: https://mlsanity.com/paper/2506.23626/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [PCGRL: Procedural Content Generation via Reinforcement Learning](https://mlsanity.com/paper/2001.09212/) (2020)
- [OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code](https://mlsanity.com/paper/2405.15568/) (2024)
- [Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning](https://mlsanity.com/paper/2008.07971/) (2020)
- [The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longer](https://mlsanity.com/paper/2502.15631/) (2025)
- [Deep Reinforcement Learning for General Video Game AI](https://mlsanity.com/paper/1806.02448/) (2018)
- [MaestroMotif: Skill Design from Artificial Intelligence Feedback](https://mlsanity.com/paper/2412.08542/) (2024)
- [Small Language Model Can Self-correct](https://mlsanity.com/paper/2401.07301/) (2024)
- ["It's Unwieldy and It Takes a Lot of Time." Challenges and Opportunities for Creating Agents in Commercial Games](https://mlsanity.com/paper/2009.00541/) (2020)
- [Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learning](https://mlsanity.com/paper/2208.07811/) (2022)
- [Technical Challenges of Deploying Reinforcement Learning Agents for Game Testing in AAA Games](https://mlsanity.com/paper/2307.11105/) (2023)
- [FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions](https://mlsanity.com/paper/2504.10002/) (2025)

## Built on by (most cited first, 1 of 1)

- [Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution](https://mlsanity.com/paper/2606.20014/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) in games has gained significant momentum in recent years, enabling the creation of different agent behaviors that can transform a player's gaming experience. However, deploying RL agents in production environments presents two key challenges: (1) designing an effective reward function typically requires an RL expert, and (2) when a game's content or mechanics are modified, previously tuned reward weights may no longer be optimal. Towards the latter challenge, we propose an automated approach for iteratively fine-tuning an RL agent's reward function weights, based on a user-defined language based behavioral goal. A Language Model (LM) proposes updated weights at each iteration based on this target behavior and a summary of performance statistics from prior training rounds. This closed-loop process allows the LM to self-correct and refine its output over time, producing increasingly aligned behavior without the need for manual reward engineering. We evaluate our approach in a racing task and show that it consistently improves agent performance across iterations. The LM-guided agents show a significant increase in performance from 9% to 74% success rate in just one iteration. We compare our LM-guided tuning against a human expert's manual weight design in the racing task: by the final iteration, the LM-tuned agent achieved an 80% success rate, and completed laps in an average of 855 time steps, a competitive performance against the expert-tuned agent's peak 94% success, and 850 time steps.