# Triangular Character Animation Sampling with Motion, Emotion, and Relation

arXiv: https://arxiv.org/abs/2203.04930 · HTML: https://mlsanity.com/paper/2203.04930/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 10 of 10)

- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Realistic Speech-Driven Facial Animation with GANs](https://mlsanity.com/paper/1906.06337/) (2019)
- [Human-centric Indoor Scene Synthesis Using Stochastic Grammar](https://mlsanity.com/paper/1808.08473/) (2018)
- [Dialogue-Based Relation Extraction](https://mlsanity.com/paper/2004.08056/) (2020)
- [Speech2Face: Learning the Face Behind a Voice](https://mlsanity.com/paper/1905.09773/) (2019)
- [Modeling Human Motion with Quaternion-based Neural Networks](https://mlsanity.com/paper/1901.07677/) (2019)
- [Learning Social Affordance for Human-Robot Interaction](https://mlsanity.com/paper/1604.03692/) (2016)
- [Skeleton-Based Online Action Prediction Using Scale Selection Network](https://mlsanity.com/paper/1902.03084/) (2019)

## Built on by (most cited first, 1 of 1)

- [Multimodal Procedural Planning via Dual Text-Image Prompting](https://mlsanity.com/paper/2305.01795/) (2023)

## Abstract (arXiv)

Dramatic progress has been made in animating individual characters. However, we still lack automatic control over activities between characters, especially those involving interactions. In this paper, we present a novel energy-based framework to sample and synthesize animations by associating the characters' body motions, facial expressions, and social relations. We propose a Spatial-Temporal And-Or graph (ST-AOG), a stochastic grammar model, to encode the contextual relationship between motion, emotion, and relation, forming a triangle in a conditional random field. We train our model from a labeled dataset of two-character interactions. Experiments demonstrate that our method can recognize the social relation between two characters and sample new scenes of vivid motion and emotion using Markov Chain Monte Carlo (MCMC) given the social relation. Thus, our method can provide animators with an automatic way to generate 3D character animations, help synthesize interactions between Non-Player Characters (NPCs), and enhance machine emotion intelligence (EQ) in virtual reality (VR).