mlsanityData to 14 Oct 2026

Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics

arXiv 2309.06687 ↗First posted 13 Sep 2023cs.RO cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

Although Deep Reinforcement Learning (DRL) has achieved notable success in numerous robotic applications, designing a high-performing reward function remains a challenging task that often requires substantial manual input. Recently, Large Language Models (LLMs) have been extensively adopted to address tasks demanding in-depth common-sense knowledge, such as reasoning and planning. Recognizing that reward function design is also inherently linked to such knowledge, LLM offers a promising potential in this context. Motivated by this, we propose in this work a novel LLM framework with a self-refinement mechanism for automated reward function design. The framework commences with the LLM formulating an initial reward function based on natural language inputs. Then, the performance of the reward function is assessed, and the results are presented back to the LLM for guiding its self-refinement process. We examine the performance of our proposed framework through a variety of continuous robotic control tasks across three diverse robotic systems. The results indicate that our LLM-designed reward functions are able to rival or even surpass manually designed reward functions, highlighting the efficacy and applicability of our approach.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

24 of 41 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  9. 2019Language Models as Knowledge Bases?cited by 1,936
  10. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  11. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  12. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
12 more
  1. 2022Large Language Models Can Self-Improvecited by 664
  2. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  3. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  4. 2023Reward Design with Language Modelscited by 261
  5. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  6. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  7. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  8. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 63
  9. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  10. 2019Evolving Rewards to Automate Reinforcement Learningcited by 31
  11. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  12. 2023Towards Building AI-CPS with NVIDIA Isaac Sim: An Industrial Benchmark and Case Study for Robotics Manipulationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 12; 2026: 6.

36 papers.

Sort:

By area:

  1. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  2. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  3. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  4. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulationcited by 45
  5. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  6. 2024EPO: Hierarchical LLM Agents with Environment Preference Optimizationcited by 28
  7. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectivescited by 27
  8. 2024Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learningcited by 23
  9. 2024CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Modelscited by 22
  10. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
  11. 2024Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Modelscited by 15
  12. 2024Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Modelscited by 10
24 more
  1. 2024LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulationcited by 9
  2. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Modelscited by 8
  3. 2025LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systemscited by 8
  4. 2024Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forwardcited by 6
  5. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  6. 2025MoE2: Optimizing Collaborative Inference for Edge Large Language Modelscited by 6
  7. 2025LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Modelscited by 4
  8. 2024LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemblecited by 3
  9. 2025PCGRLLM: Large Language Model-Driven Reward Design for Procedural Content Generation Reinforcement Learningcited by 3
  10. 2025MoColl: Agent-Based Specific and General Model Collaboration for Image Captioningcited by 2
  11. 2025Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation with Large Language Modelscited by 2
  12. 2025CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Taskscited by 2
  13. 2025VIRAL: Vision-grounded Integration for Reward design And Learningcited by 1
  14. 2025Uncertainty-aware Reward Design Processcited by 1
  15. 2026DCoPilot: Generative AI-Empowered Policy Adaptation for Dynamic Data Center Operationscited by 1
  16. 2026Novelty Adaptation Through Hybrid Large Language Model (LLM)-Symbolic Planning and LLM-guided Reinforcement Learningcited by 1
  17. 2026Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Designcited by 1
  18. 2024ARO: Large Language Model Supervised Robotics Text2Skill Autonomous Learningcited by 0
  19. 2024An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problemscited by 0
  20. 2025Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme Onecited by 0
  21. 2025Agent2: An Agent-Generates-Agent Framework for Reinforcement Learning Automationcited by 0
  22. 2026MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidancecited by 0
  23. 2026Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Datacited by 0
  24. 2026Enhancing VLM Reward Models Through Structure-Aware Fine-Tuningcited by 0