# MoFM: A Large-Scale Human Motion Foundation Model

arXiv: https://arxiv.org/abs/2502.05432 · HTML: https://mlsanity.com/paper/2502.05432/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 79 of its 104 references that are in the map.
- First posted on arXiv 8 Feb 2025.

## Builds on (most cited first, 30 of 79)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Future Frame Prediction for Anomaly Detection -- A New Baseline](https://mlsanity.com/paper/1712.09867/) (2017)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [Fine-tuned CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2212.03640/) (2022)
- [MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video](https://mlsanity.com/paper/2203.00859/) (2022)

## Built on by (most cited first, 5 of 5)

- [MoCLIP: Motion-Aware Fine-Tuning and Distillation of CLIP for Human Motion Generation](https://mlsanity.com/paper/2505.10810/) (2025)
- [Adversarially-Refined VQ-GAN with Dense Motion Tokenization for Spatio-Temporal Heatmaps](https://mlsanity.com/paper/2509.19252/) (2025)
- [DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting](https://mlsanity.com/paper/2609.08117/) (2026)
- [Cognitive State Inference from VR Motion via Motion Foundation Model](https://mlsanity.com/paper/2509.24255/) (2025)
- [World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories](https://mlsanity.com/paper/2610.01742/) (2026)

## Abstract (arXiv)

Foundation Models (FM) have increasingly drawn the attention of researchers due to their scalability and generalization across diverse tasks. Inspired by the success of FMs and the principles that have driven advancements in Large Language Models (LLMs), we introduce MoFM as a novel Motion Foundation Model. MoFM is designed for the semantic understanding of complex human motions in both time and space. To facilitate large-scale training, MotionBook, a comprehensive human motion dictionary of discretized motions is designed and employed. MotionBook utilizes Thermal Cubes to capture spatio-temporal motion heatmaps, applying principles from discrete variational models to encode human movements into discrete units for a more efficient and scalable representation. MoFM, trained on a large corpus of motion data, provides a foundational backbone adaptable to diverse downstream tasks, supporting paradigms such as one-shot, unsupervised, and supervised tasks. This versatility makes MoFM well-suited for a wide range of motion-based applications.