# Continuous Intermediate Token Learning with Implicit Motion Manifold for Keyframe Based Motion Interpolation

arXiv: https://arxiv.org/abs/2303.14926 · HTML: https://mlsanity.com/paper/2303.14926/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021)
- [Understanding the Difficulty of Training Transformers](https://mlsanity.com/paper/2004.08249/) (2020)
- [Are Transformers More Robust Than CNNs?](https://mlsanity.com/paper/2111.05464/) (2021)
- [BEVT: BERT Pretraining of Video Transformers](https://mlsanity.com/paper/2112.01529/) (2021)
- [Video Frame Interpolation Transformer](https://mlsanity.com/paper/2111.13817/) (2021)
- [MSR-GCN: Multi-Scale Residual Graph Convolution Networks for Human Motion Prediction](https://mlsanity.com/paper/2108.07152/) (2021)
- [A Neural Temporal Model for Human Motion Prediction](https://mlsanity.com/paper/1809.03036/) (2018)
- [Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fields](https://mlsanity.com/paper/2207.13807/) (2022)
- [Convolutional Autoencoders for Human Motion Infilling](https://mlsanity.com/paper/2010.11531/) (2020)
- [Progressively Generating Better Initial Guesses Towards Next Stages for High-Quality Human Motion Prediction](https://mlsanity.com/paper/2203.16051/) (2022)
- [BiHMP-GAN: Bidirectional 3D Human Motion Prediction GAN](https://mlsanity.com/paper/1812.02591/) (2018)
- [Recurrent Transition Networks for Character Locomotion](https://mlsanity.com/paper/1810.02363/) (2018)
- [Learning to Incorporate Structure Knowledge for Image Inpainting](https://mlsanity.com/paper/2002.04170/) (2020)
- [Human Motion Prediction via Learning Local Structure Representations and Temporal Dependencies](https://mlsanity.com/paper/1902.07367/) (2019)
- [ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis](https://mlsanity.com/paper/2011.10185/) (2020)
- [Diverse Human Motion Prediction via Gumbel-Softmax Sampling from an Auxiliary Space](https://mlsanity.com/paper/2207.07351/) (2022)

## Built on by (most cited first, 12 of 12)

- [KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding](https://mlsanity.com/paper/2409.01113/) (2024)
- [Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer](https://mlsanity.com/paper/2405.15439/) (2024)
- [MASIV: Toward Material-Agnostic System Identification from Videos](https://mlsanity.com/paper/2508.01112/) (2025)
- [Motion Keyframe Interpolation for Any Human Skeleton via Temporally Consistent Point Cloud Sampling and Reconstruction](https://mlsanity.com/paper/2405.07444/) (2024)
- [Scalable Group Choreography via Variational Phase Manifold Learning](https://mlsanity.com/paper/2407.18839/) (2024)
- [Generative Motion Infilling From Imprecisely Timed Keyframes](https://mlsanity.com/paper/2503.01016/) (2025)
- [Towards Synthesized and Editable Motion In-Betweening Through Part-Wise Phase Representation](https://mlsanity.com/paper/2503.08180/) (2025)
- [PUMPS: Skeleton-Agnostic Point-based Universal Motion Pre-Training for Synthesis in Human Motion Tasks](https://mlsanity.com/paper/2507.20170/) (2025)
- [SILK: Smooth InterpoLation frameworK for motion in-betweening A Simplified Computational Approach](https://mlsanity.com/paper/2506.09075/) (2025)
- [Shot Segmentation Based on Von Neumann Entropy for Key Frame Extraction](https://mlsanity.com/paper/2408.15844/) (2024)
- [Generating Detailed Character Motion from Blocking Poses](https://mlsanity.com/paper/2509.16064/) (2025)
- [Adaptive Interpolation-Synthesis for Motion In-Betweening on Keyframe-Based Animation](https://mlsanity.com/paper/2605.02742/) (2026)

## Abstract (arXiv)

Deriving sophisticated 3D motions from sparse keyframes is a particularly challenging problem, due to continuity and exceptionally skeletal precision. The action features are often derivable accurately from the full series of keyframes, and thus, leveraging the global context with transformers has been a promising data-driven embedding approach. However, existing methods are often with inputs of interpolated intermediate frame for continuity using basic interpolation methods with keyframes, which result in a trivial local minimum during training. In this paper, we propose a novel framework to formulate latent motion manifolds with keyframe-based constraints, from which the continuous nature of intermediate token representations is considered. Particularly, our proposed framework consists of two stages for identifying a latent motion subspace, i.e., a keyframe encoding stage and an intermediate token generation stage, and a subsequent motion synthesis stage to extrapolate and compose motion data from manifolds. Through our extensive experiments conducted on both the LaFAN1 and CMU Mocap datasets, our proposed method demonstrates both superior interpolation accuracy and high visual similarity to ground truth motions.