# LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music

arXiv: https://arxiv.org/abs/2606.31158 · HTML: https://mlsanity.com/paper/2606.31158/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 7 references that are in the map.
- First posted on arXiv 30 Jun 2026.

## Builds on (most cited first, 5 of 5)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [HaGRID - HAnd Gesture Recognition Image Dataset](https://mlsanity.com/paper/2206.08219/) (2022)
- [Beat this! Accurate beat tracking without DBN postprocessing](https://mlsanity.com/paper/2407.21658/) (2024)
- [Hand and Arm Gesture-based Human-Robot Interaction: A Review](https://mlsanity.com/paper/2209.08229/) (2022)
- [Quantization for OpenAI's Whisper Models: A Comparative Analysis](https://mlsanity.com/paper/2503.09905/) (2025)

## Abstract (arXiv)

The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics. Traditional methods often rely on rigid, pre-programmed commands that limit the robot's expressiveness and adaptability. This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs) to synthesize complex robotic actions from a rich tapestry of multimodal human inputs: natural speech, hand gestures, and music/sound beats. Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection. These processed inputs are contextualized using prompt templates and fed into a LLM. The LLM, informed by a predefined robot action space, reasons over the combined inputs to generate a coherent sequence of actions. This sequence is dispatched to an action queue for execution on a quadruped robot over ROS. The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music. This work represents a step towards creating robots that can interact with humans in a more fluid, creative, and context-aware manner.