# MaskControl: Spatio-Temporal Control for Masked Motion Synthesis

arXiv: https://arxiv.org/abs/2410.10780 · HTML: https://mlsanity.com/paper/2410.10780/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 54 of its 60 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 54)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [Diffusion-based Generation, Optimization, and Planning in 3D Scenes](https://mlsanity.com/paper/2301.06015/) (2023)
- [OmniControl: Control Any Joint at Any Time for Human Motion Generation](https://mlsanity.com/paper/2310.08580/) (2023)
- [Guided Motion Diffusion for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2305.12577/) (2023)
- [Perpetual Humanoid Control for Real-time Simulated Avatars](https://mlsanity.com/paper/2305.06456/) (2023)
- [FLAME: Free-form Language-based Motion Synthesis & Editing](https://mlsanity.com/paper/2209.00349/) (2022)
- [MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model](https://mlsanity.com/paper/2404.19759/) (2024)
- [Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion](https://mlsanity.com/paper/2304.01893/) (2023)

## Built on by (most cited first, 30 of 38)

- [Kimodo: Scaling Controllable Human Motion Generation](https://mlsanity.com/paper/2603.15546/) (2026)
- [HY-Motion 1.0: Scaling Flow Matching Models for Text-To-Motion Generation](https://mlsanity.com/paper/2512.23464/) (2025)
- [Absolute Coordinates Make Motion Generation Easy](https://mlsanity.com/paper/2505.19377/) (2025)
- [HERO: Human Reaction Generation from Videos](https://mlsanity.com/paper/2503.08270/) (2025)
- [ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation](https://mlsanity.com/paper/2607.08741/) (2026)
- [UniAct: Unified Motion Generation and Action Streaming for Humanoid Robots](https://mlsanity.com/paper/2512.24321/) (2025)
- [ActionPlan: Future-Aware Streaming Motion Synthesis via Frame-Level Action Planning](https://mlsanity.com/paper/2603.13500/) (2026)
- [UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors](https://mlsanity.com/paper/2603.15975/) (2026)
- [MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation](https://mlsanity.com/paper/2601.19577/) (2026)
- [Bridging Semantic and Kinematic Conditions with Diffusion-based Discrete Motion Tokenizer](https://mlsanity.com/paper/2603.19227/) (2026)
- [MotionHiFlow: Text-to-motion via hierarchical flow matching](https://mlsanity.com/paper/2604.23264/) (2026)
- [MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation](https://mlsanity.com/paper/2512.13840/) (2025)
- [PhysMoDPO: Physically-Plausible Humanoid Motion with Preference Optimization](https://mlsanity.com/paper/2603.13228/) (2026)
- [HOIMask: Towards Generative Masked Modeling for Human Object Interaction Generation](https://mlsanity.com/paper/2608.15141/) (2026)
- [Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward](https://mlsanity.com/paper/2506.03191/) (2025)
- [Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reach](https://mlsanity.com/paper/2512.16456/) (2025)
- [T2MBench: A Benchmark for Out-of-Distribution Text-to-Motion Generation](https://mlsanity.com/paper/2602.13751/) (2026)
- [DreamControl-v2: Simpler and Scalable Autonomous Humanoid Skills via Trainable Guided Diffusion Priors](https://mlsanity.com/paper/2604.00202/) (2026)
- [AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling](https://mlsanity.com/paper/2605.29488/) (2026)
- [KV-Control: Parameter-Efficient K/V Injection for Trajectory-Controlled Text-to-Motion](https://mlsanity.com/paper/2606.05624/) (2026)
- [Generating Detailed Character Motion from Blocking Poses](https://mlsanity.com/paper/2509.16064/) (2025)
- [Pressure2Motion: Hierarchical Human Motion Reconstruction from Ground Pressure with Text Guidance](https://mlsanity.com/paper/2511.05038/) (2025)
- [Controllable Text-to-Motion Generation via Modular Body-Part Phase Control](https://mlsanity.com/paper/2603.19795/) (2026)
- [NOUS: Video-Driven 3D Human Reaction Generation via Observation-Reaction Mutual Steering](https://mlsanity.com/paper/2603.20187/) (2026)
- [Unified Number-Free Text-to-Motion Generation Via Flow Matching](https://mlsanity.com/paper/2603.27040/) (2026)
- [Exploring Motion-Language Alignment for Text-driven Motion Generation](https://mlsanity.com/paper/2604.02973/) (2026)
- [Towards Highly-Constrained Human Motion Generation with Retrieval-Guided Diffusion Noise Optimization](https://mlsanity.com/paper/2605.08054/) (2026)
- [EgoForce: Robust Online Egocentric Motion Reconstruction via Diffusion Forcing](https://mlsanity.com/paper/2605.13041/) (2026)
- [SCAMP: Sparse-anchor Control is One Small Projection](https://mlsanity.com/paper/2605.14716/) (2026)
- [Multi-scale Coarse-to-fine Modeling for Test-time Human Motion Control](https://mlsanity.com/paper/2605.14935/) (2026)

## Abstract (arXiv)

Recent advances in motion diffusion models have enabled spatially controllable text-to-motion generation. However, these models struggle to achieve high-precision control while maintaining high-quality motion generation. To address these challenges, we propose MaskControl, the first approach to introduce controllability to the generative masked motion model. Our approach introduces two key innovations. First, textitLogits Regularizer implicitly perturbs logits at training time to align the distribution of motion tokens with the controlled joint positions, while regularizing the categorical token prediction to ensure high-fidelity generation. Second, textitLogit Optimization explicitly optimizes the predicted logits during inference time, directly reshaping the token distribution that forces the generated motion to accurately align with the controlled joint positions. Moreover, we introduce textitDifferentiable Expectation Sampling (DES) to combat the non-differential distribution sampling process encountered by logits regularizer and optimization. Extensive experiments demonstrate that MaskControl outperforms state-of-the-art methods, achieving superior motion quality (FID decreases by ~77%) and higher control precision (average error 0.91 vs. 1.08). Additionally, MaskControl enables diverse applications, including any-joint-any-frame control, body-part timeline control, and zero-shot objective control. Video visualization can be found at https://www.ekkasit.com/ControlMM-page/