# CALM: Conditional Adversarial Latent Models for Directable Virtual Characters

arXiv: https://arxiv.org/abs/2305.02195 · HTML: https://mlsanity.com/paper/2305.02195/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [NormFace: L2 Hypersphere Embedding for Face Verification](https://mlsanity.com/paper/1704.06369/) (2017)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Character Controllers Using Motion VAEs](https://mlsanity.com/paper/2103.14274/) (2021)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [Unsupervised Learning by Predicting Noise](https://mlsanity.com/paper/1704.05310/) (2017)
- [A Deep Hierarchical Approach to Lifelong Learning in Minecraft](https://mlsanity.com/paper/1604.07255/) (2016)
- [Spherical Latent Spaces for Stable Variational Autoencoders](https://mlsanity.com/paper/1808.10805/) (2018)
- [PADL: Language-Directed Physics-Based Character Control](https://mlsanity.com/paper/2301.13868/) (2023)

## Built on by (most cited first, 30 of 94)

- [Open-TeleVision: Teleoperation with Immersive Active Visual Feedback](https://mlsanity.com/paper/2407.01512/) (2024)
- [Expressive Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2402.16796/) (2024)
- [Universal Humanoid Motion Representations for Physics-Based Control](https://mlsanity.com/paper/2310.04582/) (2023)
- [MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting](https://mlsanity.com/paper/2409.14393/) (2024)
- [Synthesizing Diverse Human Motions in 3D Indoor Scenes](https://mlsanity.com/paper/2305.12411/) (2023)
- [SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control](https://mlsanity.com/paper/2511.07820/) (2025)
- [TLControl: Trajectory and Language Control for Human Motion Synthesis](https://mlsanity.com/paper/2311.17135/) (2023)
- [EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generation](https://mlsanity.com/paper/2312.02256/) (2023)
- [ExBody2: Advanced Expressive Humanoid Whole-Body Control](https://mlsanity.com/paper/2412.13196/) (2024)
- [InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2502.20390/) (2025)
- [Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives](https://mlsanity.com/paper/2403.10518/) (2024)
- [MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations](https://mlsanity.com/paper/2310.10198/) (2023)
- [Taming Diffusion Probabilistic Models for Character Control](https://mlsanity.com/paper/2404.15121/) (2024)
- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [Interactive Character Control with Auto-Regressive Motion Diffusion Models](https://mlsanity.com/paper/2306.00416/) (2023)
- [C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters](https://mlsanity.com/paper/2309.11351/) (2023)
- [Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Control](https://mlsanity.com/paper/2412.07773/) (2024)
- [AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Control](https://mlsanity.com/paper/2505.03738/) (2025)
- [Synthesizing Physically Plausible Human Motions in 3D Scenes](https://mlsanity.com/paper/2308.09036/) (2023)
- [CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics](https://mlsanity.com/paper/2406.14558/) (2024)
- [Iterative Motion Editing with Natural Language](https://mlsanity.com/paper/2312.11538/) (2023)
- [Neural Categorical Priors for Physics-Based Character Control](https://mlsanity.com/paper/2308.07200/) (2023)
- [AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents](https://mlsanity.com/paper/2403.12835/) (2024)
- [Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning](https://mlsanity.com/paper/2309.11359/) (2023)
- [SkillMimic: Learning Basketball Interaction Skills from Demonstrations](https://mlsanity.com/paper/2408.15270/) (2024)
- [Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policies](https://mlsanity.com/paper/2410.11825/) (2024)
- [SMPLOlympics: Sports Environments for Physically Simulated Humanoids](https://mlsanity.com/paper/2407.00187/) (2024)
- [Learning from Massive Human Videos for Universal Humanoid Pose Control](https://mlsanity.com/paper/2412.14172/) (2024)
- [Revisit Human-Scene Interaction via Space Occupancy](https://mlsanity.com/paper/2312.02700/) (2023)
- [UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Control](https://mlsanity.com/paper/2504.12540/) (2025)

## Abstract (arXiv)

In this work, we present Conditional Adversarial Latent Models (CALM), an approach for generating diverse and directable behaviors for user-controlled interactive virtual characters. Using imitation learning, CALM learns a representation of movement that captures the complexity and diversity of human motion, and enables direct control over character movements. The approach jointly learns a control policy and a motion encoder that reconstructs key characteristics of a given motion without merely replicating it. The results show that CALM learns a semantic motion representation, enabling control over the generated motions and style-conditioning for higher-level task training. Once trained, the character can be controlled using intuitive interfaces, akin to those found in video games.