mlsanityData to 7 Oct 2026

Learning Reward for Robot Skills Using Large Language Models via Self-Alignment

arXiv 2405.07162 ↗First posted 12 May 2024cs.RO cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Learning reward functions remains the bottleneck to equip a robot with a broad repertoire of skills. Large Language Models (LLM) contain valuable task-related knowledge that can potentially aid in the learning of reward functions. However, the proposed reward function can be imprecise, thus ineffective which requires to be further grounded with environment information. We proposed a method to learn rewards more efficiently in the absence of humans. Our approach consists of two components: We first use the LLM to propose features and parameterization of the reward, then update the parameters through an iterative self-alignment process. In particular, the process minimizes the ranking inconsistency between the LLM and the learnt reward functions based on the execution feedback. The method was validated on 9 tasks across 2 simulation environments. It demonstrates a consistent improvement over training efficacy and efficiency, meanwhile consuming significantly fewer GPT tokens compared to the alternative mutation-based method.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 27 of its 52 references that are in the map. First posted on arXiv 12 May 2024.

Builds on

27 of 52 references.

References · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  8. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  9. 2019Solving Rubik's Cube with a Robot Handcited by 699
  10. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  11. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  12. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
15 more
  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  2. 2023Reward Design with Language Modelscited by 257
  3. 2017Inverse Reward Designcited by 246
  4. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  5. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  6. 2019Imitation Learning as f-Divergence Minimizationcited by 121
  7. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 92
  8. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  9. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  10. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  11. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  12. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 42
  13. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 18
  14. 2021APReL: A Library for Active Preference-based Reward Learning Algorithmscited by 14
  15. 2023Reward Learning with Intractable Normalizing Functionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 4.

19 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  3. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  4. 2024DAG-Plan: Generating Directed Acyclic Dependency Graphs for Dual-Arm Cooperative Planningcited by 14
  5. 2024GRS: Generating Robotic Simulation Tasks from Real-World Imagescited by 13
  6. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learningcited by 10
  7. 2024LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classificationcited by 6
  8. 2025ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Taskscited by 6
  9. 2025Reward Evolution with Graph-of-Thoughts: A Bi-Level Language Model Framework for Reinforcement Learningcited by 6
  10. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  11. 2025RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skillscited by 4
  12. 2024Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcementcited by 3
7 more
  1. 2025Adaptive Articulated Object Manipulation On The Fly with Foundation Model Reasoning and Part Groundingcited by 3
  2. 2026RF-Agent: Automated Reward Function Design via Language Agent Tree Searchcited by 3
  3. 2025Boosting Universal LLM Reward Design through Heuristic Reward Observation Space Evolutioncited by 2
  4. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learningcited by 2
  5. 2025LaMOuR: Leveraging Language Models for Out-of-Distribution Recovery in Reinforcement Learningcited by 1
  6. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learningcited by 0
  7. 2026Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Learning Reward for Physical Skills using Large Language Model
  2. 2024Self-Rewarding Language Models
  3. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  4. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learning
  5. 2024Agentic Skill Discovery
  6. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs
  7. 2025Generalist Reward Models: Found Inside Large Language Models
  8. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models