# EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold

arXiv: https://arxiv.org/abs/2609.35616 · HTML: https://mlsanity.com/paper/2609.35616/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 30 of 42)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time](https://mlsanity.com/paper/2404.10667/) (2024)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [Test-Time Training Done Right](https://mlsanity.com/paper/2505.23884/) (2025)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models](https://mlsanity.com/paper/2310.00434/) (2023)
- [Responsive Listening Head Generation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2112.13548/) (2021)
- [PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models](https://mlsanity.com/paper/2602.06053/) (2026)
- [Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videos](https://mlsanity.com/paper/2207.11094/) (2022)
- [INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations](https://mlsanity.com/paper/2412.04037/) (2024)
- [Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset](https://mlsanity.com/paper/2506.22554/) (2025)
- [Affective Faces for Goal-Driven Dyadic Communication](https://mlsanity.com/paper/2301.10939/) (2023)
- [Dyadic Interaction Modeling for Social Behavior Generation](https://mlsanity.com/paper/2403.09069/) (2024)
- [DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations](https://mlsanity.com/paper/2505.18096/) (2025)
- [ARTalk: Speech-Driven 3D Head Animation via Autoregressive Model](https://mlsanity.com/paper/2502.20323/) (2025)
- [RoboTTT: Context Scaling for Robot Policies](https://mlsanity.com/paper/2607.15275/) (2026)
- [ViT3: Unlocking Test-Time Training in Vision](https://mlsanity.com/paper/2512.01643/) (2025)

## Abstract (arXiv)

Interactive 3D head generation requires coordinated speaking and listening motion that responds to an evolving conversation. Existing generators use incoming observations as context but keep their parameters fixed, leaving conversational patterns unused as a learning signal. We introduce EvolvingAvatar, a causal generator that uses test-time training to adapt to user face video and dyadic audio during interaction. Its dyadic context prediction objective provides a self-supervised learning signal from audiovisual context without target motion labels at test time. Persistent fast weights accumulate these updates within each conversation to guide motion generation, while transient jaw adaptation responds to current audiovisual context. Predicted speech activity controls how persistent adaptation guides motion. We also introduce InterHead-Bench, a unified 455.95-hour benchmark built from single-view and dual-view conversation videos. Experiments show improved conversational motion statistics over strong baselines. On the hardest out-of-distribution split, generation improves as conversations unfold, reducing mismatch with recorded user-avatar expression statistics by up to 11.1% from the first interval.