# A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robot

arXiv: https://arxiv.org/abs/2603.21013 · HTML: https://mlsanity.com/paper/2603.21013/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 37 references that are in the map.
- First posted on arXiv 9 Jan 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 10 of 10)

- [Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction](https://mlsanity.com/paper/2508.05294/) (2025)
- [MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols](https://mlsanity.com/paper/2508.18240/) (2025)
- [A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment](https://mlsanity.com/paper/2503.06241/) (2025)
- [A Sign Language Recognition System with Pepper, Lightweight-Transformer, and LLM](https://mlsanity.com/paper/2309.16898/) (2023)
- [I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots](https://mlsanity.com/paper/2311.08957/) (2023)
- [Direct Speech to Speech Translation: A Review](https://mlsanity.com/paper/2503.04799/) (2025)
- [Does ChatGPT and Whisper Make Humanoid Robots More Relatable?](https://mlsanity.com/paper/2402.07095/) (2024)
- [Towards Multimodal Social Conversations with Robots: Using Vision-Language Models](https://mlsanity.com/paper/2507.19196/) (2025)
- [Touch in Human Social Robot Interaction: Systematic Literature Review with PRISMA Method](https://mlsanity.com/paper/2407.11834/) (2024)
- [Personalized Socially Assistive Robots With End-to-End Speech-Language Models For Well-Being Support](https://mlsanity.com/paper/2507.14412/) (2025)

## Built on by (most cited first, 1 of 1)

- [Teachy Mini: Development and Preliminary Evaluation of a Knowledge-Based Generative Social Robot for Higher Education](https://mlsanity.com/paper/2607.22345/) (2026)

## Abstract (arXiv)

Despite recent advances in integrating Large Language Models (LLMs) into social robotics, two weaknesses persist. First, existing implementations on platforms like Pepper often rely on cascaded Speech-to-Text (STT)->LLM->Text-to-Speech (TTS) pipelines, resulting in high latency and the loss of paralinguistic information. Second, most implementations fail to fully leverage the LLM's capabilities for multimodal perception and agentic control. We present an open-source Android framework for the Pepper robot that addresses these limitations through two key innovations. First, we integrate end-to-end Speech-to-Speech (S2S) models to achieve low-latency interaction while preserving paralinguistic cues and enabling adaptive intonation. Second, we implement extensive Function Calling capabilities that elevate the LLM to an agentic planner, orchestrating robot actions (navigation, gaze control, tablet interaction) and integrating diverse multimodal feedback (vision, touch, system state). The framework runs on the robot's tablet but can also be built to run on regular Android smartphones or tablets, decoupling development from robot hardware. This work provides the HRI community with a practical, extensible platform for exploring advanced LLM-driven embodied interaction.