# AMII: Adaptive Multimodal Inter-personal and Intra-personal Model for Adapted Behavior Synthesis

arXiv: https://arxiv.org/abs/2305.11310 · HTML: https://mlsanity.com/paper/2305.11310/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 35 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 5 of 5)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents](https://mlsanity.com/paper/2101.11101/) (2021)
- [Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings](https://mlsanity.com/paper/2006.09888/) (2020)

## Built on by (most cited first, 6 of 6)

- [Dyadic Interaction Modeling for Social Behavior Generation](https://mlsanity.com/paper/2403.09069/) (2024)
- [Reversible Graph Neural Network-based Reaction Distribution Learning for Multiple Appropriate Facial Reactions Generation](https://mlsanity.com/paper/2305.15270/) (2023)
- [REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2306.06583/) (2023)
- [ReactDiff: Fundamental Multiple Appropriate Facial Reaction Diffusion Model](https://mlsanity.com/paper/2510.04712/) (2025)
- [ComFace: Facial Representation Learning with Synthetic Data for Comparing Faces](https://mlsanity.com/paper/2405.16016/) (2024)
- [ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation](https://mlsanity.com/paper/2609.33749/) (2026)

## Abstract (arXiv)

Socially Interactive Agents (SIAs) are physical or virtual embodied agents that display similar behavior as human multimodal behavior. Modeling SIAs' non-verbal behavior, such as speech and facial gestures, has always been a challenging task, given that a SIA can take the role of a speaker or a listener. A SIA must emit appropriate behavior adapted to its own speech, its previous behaviors (intra-personal), and the User's behaviors (inter-personal) for both roles. We propose AMII, a novel approach to synthesize adaptive facial gestures for SIAs while interacting with Users and acting interchangeably as a speaker or as a listener. AMII is characterized by modality memory encoding schema - where modality corresponds to either speech or facial gestures - and makes use of attention mechanisms to capture the intra-personal and inter-personal relationships. We validate our approach by conducting objective evaluations and comparing it with the state-of-the-art approaches.