# Ain't Misbehavin' -- Using LLMs to Generate Expressive Robot Behavior in Conversations with the Tabletop Robot Haru

arXiv: https://arxiv.org/abs/2402.11571 · HTML: https://mlsanity.com/paper/2402.11571/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Builds on (most cited first, 8 of 8)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Large Language Models for Robotics: A Survey](https://mlsanity.com/paper/2311.07226/) (2023)
- [Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data](https://mlsanity.com/paper/2301.05843/) (2023)
- [Using Socially Expressive Mixed Reality Arms for Enhancing Low-Expressivity Robots](https://mlsanity.com/paper/1911.09713/) (2019)

## Built on by (most cited first, 5 of 5)

- [EMOTION: Expressive Motion Sequence Generation for Humanoid Robots with In-Context Learning](https://mlsanity.com/paper/2410.23234/) (2024)
- [Empathic Grounding: Explorations using Multimodal Interaction and Large Language Models with Conversational Agents](https://mlsanity.com/paper/2407.01824/) (2024)
- [The Role of Generative AI in Facilitating Social Interactions: A Scoping Review](https://mlsanity.com/paper/2506.10927/) (2025)
- [Robot guide with multi-agent control and automatic scenario generation with LLM](https://mlsanity.com/paper/2509.10317/) (2025)
- [How Should I Pick a Foundation Model for My Robot? In Favor of a Community Evaluation Framework for Social Robots](https://mlsanity.com/paper/2608.06898/) (2026)

## Abstract (arXiv)

Social robots aim to establish long-term bonds with humans through engaging conversation. However, traditional conversational approaches, reliant on scripted interactions, often fall short in maintaining engaging conversations. This paper addresses this limitation by integrating large language models (LLMs) into social robots to achieve more dynamic and expressive conversations. We introduce a fully-automated conversation system that leverages LLMs to generate robot responses with expressive behaviors, congruent with the robot's personality. We incorporate robot behavior with two modalities: 1) a text-to-speech (TTS) engine capable of various delivery styles, and 2) a library of physical actions for the robot. We develop a custom, state-of-the-art emotion recognition model to dynamically select the robot's tone of voice and utilize emojis from LLM output as cues for generating robot actions. A demo of our system is available here. To illuminate design and implementation issues, we conduct a pilot study where volunteers chat with a social robot using our proposed system, and we analyze their feedback, conducting a rigorous error analysis of chat transcripts. Feedback was overwhelmingly positive, with participants commenting on the robot's empathy, helpfulness, naturalness, and entertainment. Most negative feedback was due to automatic speech recognition (ASR) errors which had limited impact on conversations. However, we observed a small class of errors, such as the LLM repeating itself or hallucinating fictitious information and human responses, that have the potential to derail conversations, raising important issues for LLM application.