# Vid2Game: Controllable Characters Extracted from Real-World Videos

arXiv: https://arxiv.org/abs/1904.08379 · HTML: https://mlsanity.com/paper/1904.08379/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2019.

## Built on by (most cited first, 18 of 18)

- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Editable Free-viewpoint Video Using a Layered Neural Representation](https://mlsanity.com/paper/2104.14786/) (2021)
- [DisCo: Disentangled Control for Realistic Human Dance Generation](https://mlsanity.com/paper/2307.00040/) (2023)
- [Make It Move: Controllable Image-to-Video Generation with Text Descriptions](https://mlsanity.com/paper/2112.02815/) (2021)
- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [Playable Video Generation](https://mlsanity.com/paper/2101.12195/) (2021)
- [Behavior-Driven Synthesis of Human Dynamics](https://mlsanity.com/paper/2103.04677/) (2021)
- [Strumming to the Beat: Audio-Conditioned Contrastive Video Textures](https://mlsanity.com/paper/2104.02687/) (2021)
- [Smile Like You Mean It: Driving Animatronic Robotic Face with Learned Models](https://mlsanity.com/paper/2105.12724/) (2021)
- [Flow Guided Transformable Bottleneck Networks for Motion Retargeting](https://mlsanity.com/paper/2106.07771/) (2021)
- [Fast-Vid2Vid: Spatial-Temporal Compression for Video-to-Video Synthesis](https://mlsanity.com/paper/2207.05049/) (2022)
- [Progressive and Aligned Pose Attention Transfer for Person Image Generation](https://mlsanity.com/paper/2103.11622/) (2021)
- [Controllable Video Generation through Global and Local Motion Dynamics](https://mlsanity.com/paper/2204.06558/) (2022)
- [Render In-between: Motion Guided Video Synthesis for Action Interpolation](https://mlsanity.com/paper/2111.01029/) (2021)
- [Direct Dense Pose Estimation](https://mlsanity.com/paper/2204.01263/) (2022)
- [MetaDance: Few-shot Dancing Video Retargeting via Temporal-aware Meta-learning](https://mlsanity.com/paper/2201.04851/) (2022)
- [Recurrent Transformer Variational Autoencoders for Multi-Action Motion Synthesis](https://mlsanity.com/paper/2206.06741/) (2022)
- [Signs of Language: Embodied Sign Language Fingerspelling Acquisition from Demonstrations for Human-Robot Interaction](https://mlsanity.com/paper/2209.05135/) (2022)

## Abstract (arXiv)

We are given a video of a person performing a certain activity, from which we extract a controllable model. The model generates novel image sequences of that person, according to arbitrary user-defined control signals, typically marking the displacement of the moving body. The generated video can have an arbitrary background, and effectively capture both the dynamics and appearance of the person. The method is based on two networks. The first network maps a current pose, and a single-instance control signal to the next pose. The second network maps the current pose, the new pose, and a given background, to an output frame. Both networks include multiple novelties that enable high-quality performance. This is demonstrated on multiple characters extracted from various videos of dancers and athletes.