# SARAH: Spatially Aware Real-time Agentic Humans

arXiv: https://arxiv.org/abs/2602.18432 · HTML: https://mlsanity.com/paper/2602.18432/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 20 Feb 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 29 of 29)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents](https://mlsanity.com/paper/2303.14613/) (2023)
- [Diffusion Adversarial Post-Training for One-Step Video Generation](https://mlsanity.com/paper/2501.08316/) (2025)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023)
- [Gesticulator: A framework for semantically-aware speech-driven gesture generation](https://mlsanity.com/paper/2001.09326/) (2020)
- [ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech](https://mlsanity.com/paper/2209.07556/) (2022)
- [DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control](https://mlsanity.com/paper/2410.05260/) (2024)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [Driving-Signal Aware Full-Body Avatars](https://mlsanity.com/paper/2105.10441/) (2021)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [Taming Diffusion Probabilistic Models for Character Control](https://mlsanity.com/paper/2404.15121/) (2024)
- [Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors](https://mlsanity.com/paper/2212.04248/) (2022)
- [Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction](https://mlsanity.com/paper/1906.04158/) (2019)
- [StreamDiT: Real-Time Streaming Text-to-Video Generation](https://mlsanity.com/paper/2507.03745/) (2025)
- [Social Scene Understanding: End-to-End Multi-Person Action Localization and Collective Activity Recognition](https://mlsanity.com/paper/1611.09078/) (2016)
- [LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation](https://mlsanity.com/paper/2309.09294/) (2023)
- [Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis](https://mlsanity.com/paper/2405.09814/) (2024)
- [TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation](https://mlsanity.com/paper/2410.04221/) (2024)
- [To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations](https://mlsanity.com/paper/1910.02181/) (2019)
- [SIGGesture: Generalized Co-Speech Gesture Synthesis via Semantic Injection with Large-Scale Pre-Training Diffusion Models](https://mlsanity.com/paper/2405.13336/) (2024)
- [Pedestrian Trajectory Prediction Based on Social Interactions Learning With Random Weights](https://mlsanity.com/paper/2501.07711/) (2025)

## Built on by (most cited first, 2 of 2)

- [UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors](https://mlsanity.com/paper/2603.15975/) (2026)
- [StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring](https://mlsanity.com/paper/2608.01643/) (2026)

## Abstract (arXiv)

As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see https://evonneng.github.io/sarah/ for details.