# Developing Social Robots with Empathetic Non-Verbal Cues Using Large Language Models

arXiv: https://arxiv.org/abs/2308.16529 · HTML: https://mlsanity.com/paper/2308.16529/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 23 references that are in the map.
- First posted on arXiv 31 Aug 2023.

## Builds on (most cited first, 6 of 6)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Agents facilitate one category of human empathy through task difficulty](https://mlsanity.com/paper/2206.06128/) (2022)

## Built on by (most cited first, 12 of 12)

- [LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions](https://mlsanity.com/paper/2406.08824/) (2024)
- [Scaffolding Empathy: Training Counselors with Simulated Patients and Utterance-level Performance Visualizations](https://mlsanity.com/paper/2502.18673/) (2025)
- [Exploring Large Language Models to Facilitate Variable Autonomy for Human-Robot Teaming](https://mlsanity.com/paper/2312.07214/) (2023)
- [Towards Embedding Dynamic Personas in Interactive Robots: Masquerading Animated Social Kinematics (MASK)](https://mlsanity.com/paper/2403.10041/) (2024)
- [Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task](https://mlsanity.com/paper/2404.08424/) (2024)
- [Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models](https://mlsanity.com/paper/2503.15491/) (2025)
- [Plan-and-Act using Large Language Models for Interactive Agreement](https://mlsanity.com/paper/2504.01252/) (2025)
- [QUB-PHEO: A Visual-Based Dyadic Multi-View Dataset for Intention Inference in Collaborative Assembly](https://mlsanity.com/paper/2409.15560/) (2024)
- [Leveraging LLMs with Iterative Loop Structure for Enhanced Social Intelligence in Video Question Answering](https://mlsanity.com/paper/2503.21190/) (2025)
- [Robot guide with multi-agent control and automatic scenario generation with LLM](https://mlsanity.com/paper/2509.10317/) (2025)
- [Designing for Disagreement: Front-End Guardrails for Assistance Allocation in LLM-Enabled Robots](https://mlsanity.com/paper/2603.16537/) (2026)
- [Closing the Affective Loop: Multimodal Speaker-Listener Emotion-Dynamics-Aware Empathetic Social Robots](https://mlsanity.com/paper/2608.16686/) (2026)

## Abstract (arXiv)

We propose augmenting the empathetic capacities of social robots by integrating non-verbal cues. Our primary contribution is the design and labeling of four types of empathetic non-verbal cues, abbreviated as SAFE: Speech, Action (gesture), Facial expression, and Emotion, in a social robot. These cues are generated using a Large Language Model (LLM). We developed an LLM-based conversational system for the robot and assessed its alignment with social cues as defined by human counselors. Preliminary results show distinct patterns in the robot's responses, such as a preference for calm and positive social emotions like 'joy' and 'lively', and frequent nodding gestures. Despite these tendencies, our approach has led to the development of a social robot capable of context-aware and more authentic interactions. Our work lays the groundwork for future studies on human-robot interactions, emphasizing the essential role of both verbal and non-verbal cues in creating social and empathetic robots.