# Generating Natural and Expressive Robot Gestures through Iterative Reinforcement Learning with Human Feedback using LLMs

arXiv: https://arxiv.org/abs/2606.18747 · HTML: https://mlsanity.com/paper/2606.18747/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [Large Language Models for Robotics: A Survey](https://mlsanity.com/paper/2311.07226/) (2023)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [Large Language Models for Robotics: Opportunities, Challenges, and Perspectives](https://mlsanity.com/paper/2401.04334/) (2024)
- [Reinforcement Learning Enhanced LLMs: A Survey](https://mlsanity.com/paper/2412.10400/) (2024)
- [Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback](https://mlsanity.com/paper/2108.05382/) (2021)
- [i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loops](https://mlsanity.com/paper/2207.06572/) (2022)
- [Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language](https://mlsanity.com/paper/2311.14543/) (2023)
- [Generative Expressive Robot Behaviors using Large Language Models](https://mlsanity.com/paper/2401.14673/) (2024)
- [Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models](https://mlsanity.com/paper/2309.04316/) (2023)
- [Reinforcement Learning Approaches in Social Robotics](https://mlsanity.com/paper/2009.09689/) (2020)
- [Generative Artificial Intelligence in Robotic Manipulation: A Survey](https://mlsanity.com/paper/2503.03464/) (2025)
- [Primitive Skill-based Robot Learning from Human Evaluative Feedback](https://mlsanity.com/paper/2307.15801/) (2023)
- [Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback](https://mlsanity.com/paper/2310.20608/) (2023)
- [Deep Reinforcement Learning with Interactive Feedback in a Human-Robot Environment](https://mlsanity.com/paper/2007.03363/) (2020)
- [Co-Speech Gesture Synthesis using Discrete Gesture Token Learning](https://mlsanity.com/paper/2303.12822/) (2023)
- [EMOTION: Expressive Motion Sequence Generation for Humanoid Robots with In-Context Learning](https://mlsanity.com/paper/2410.23234/) (2024)
- [The Impact of VR and 2D Interfaces on Human Feedback in Preference-Based Robot Learning](https://mlsanity.com/paper/2503.16500/) (2025)
- [Would you let a humanoid play storytelling with your child? A usability study on LLM-powered narrative Human-Robot Interaction](https://mlsanity.com/paper/2508.02505/) (2025)

## Abstract (arXiv)

Expressive gestures are essential for natural and effective communication, complementing speech when verbal cues alone are insufficient (e.g., pointing). For social robots such as the humanoid Pepper, producing natural and expressive movements is critical for improving human-robot interaction (HRI) and long-term acceptance. However, generating gestures remains challenging due to reliance on expert-authored animations, resulting in rigid behaviors that are impractical for dynamic and diverse environments. Alternatively, machine learning approaches often struggle to capture perceived naturalness, becoming increasingly challenging with more degrees of freedom. Consequently, producing expressive robot gestures requires a system that can adapt to the environment while adhering to social norms and physical constraints. Recent advances in large language models (LLMs) enable dynamic code generation, offering new opportunities for runtime gesture synthesis from natural language. In this paper, we integrate ChatGPT into the humanoid robot Pepper to generate co-speech gestures aligned with conversational output. While this baseline enables flexible gesture generation, the resulting motions are often perceived as stiff and unnatural. To address this limitation, we introduce an iterative reinforcement learning with human feedback (RLHF) system that finetunes gesture generation based on user evaluations, leveraging an iterative user study to compare Pepper's generated gestures. Our results show that RLHF improved the LLM's co-speech generative capabilities, producing more expressive, relevant and fluid movements.