# Human-Centric Foundation Models: Perception, Generation and Agentic Modeling

arXiv: https://arxiv.org/abs/2502.08556 · HTML: https://mlsanity.com/paper/2502.08556/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 50 of its 56 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 50)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset](https://mlsanity.com/paper/2307.00818/) (2023)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation](https://mlsanity.com/paper/2309.17448/) (2023)
- [MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators](https://mlsanity.com/paper/2306.10900/) (2023)
- [StyleGAN-Human: A Data-Centric Odyssey of Human Generation](https://mlsanity.com/paper/2204.11823/) (2022)
- [OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models](https://mlsanity.com/paper/2502.01061/) (2025)
- [HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation](https://mlsanity.com/paper/2304.04269/) (2023)
- [DreamHuman: Animatable 3D Avatars from Text](https://mlsanity.com/paper/2306.09329/) (2023)
- [Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks](https://mlsanity.com/paper/2303.17602/) (2023)
- [ChatPose: Chatting about 3D Human Pose](https://mlsanity.com/paper/2311.18836/) (2023)
- [HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion](https://mlsanity.com/paper/2310.08579/) (2023)
- [CoMo: Controllable Motion Generation through Language Guided Pose Code Editing](https://mlsanity.com/paper/2403.13900/) (2024)
- [SofGAN: A Portrait Image Generator with Dynamic Styling](https://mlsanity.com/paper/2007.03780/) (2020)
- [AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyond](https://mlsanity.com/paper/2311.16468/) (2023)
- [Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformer](https://mlsanity.com/paper/2405.17405/) (2024)
- [UniHCP: A Unified Model for Human-Centric Perceptions](https://mlsanity.com/paper/2303.02936/) (2023)
- [MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding](https://mlsanity.com/paper/2410.21747/) (2024)
- [AG3D: Learning to Generate 3D Avatars from 2D Image Collections](https://mlsanity.com/paper/2305.02312/) (2023)
- [Instruct-ReID: A Multi-purpose Person Re-identification Task with Instructions](https://mlsanity.com/paper/2306.07520/) (2023)
- [The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion](https://mlsanity.com/paper/2412.10523/) (2024)
- [HumanBench: Towards General Human-centric Perception with Projector Assisted Pretraining](https://mlsanity.com/paper/2303.05675/) (2023)
- [M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation](https://mlsanity.com/paper/2405.16273/) (2024)

## Built on by (most cited first, 12 of 12)

- [Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation](https://mlsanity.com/paper/2504.02542/) (2025)
- [MMGen: Unified Multi-modal Image Generation and Understanding in One Go](https://mlsanity.com/paper/2503.20644/) (2025)
- [VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation](https://mlsanity.com/paper/2604.10127/) (2026)
- [Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams](https://mlsanity.com/paper/2608.12262/) (2026)
- [MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation](https://mlsanity.com/paper/2507.05092/) (2025)
- [HUG-VIS: A Multimodal Benchmark for Human-centered Understanding and Generation in Visual Intelligence](https://mlsanity.com/paper/2608.26517/) (2026)
- [FMGS-Avatar: Mesh-Guided 2D Gaussian Splatting with Foundation Model Priors for 3D Monocular Avatar Reconstruction](https://mlsanity.com/paper/2509.14739/) (2025)
- [The Global-Local loop: what is missing in bridging the gap between geospatial data from numerous communities?](https://mlsanity.com/paper/2603.20305/) (2026)
- [Marrying Text-to-Motion Generation with Skeleton-Based Action Recognition](https://mlsanity.com/paper/2604.17090/) (2026)
- [Human-Centric Intelligence in the Era of Foundation Models: A Survey](https://mlsanity.com/paper/2608.18184/) (2026)
- [Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates](https://mlsanity.com/paper/2608.21160/) (2026)
- [VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Quality](https://mlsanity.com/paper/2608.25452/) (2026)

## Abstract (arXiv)

Human understanding and generation are critical for modeling digital humans and humanoid embodiments. Recently, Human-centric Foundation Models (HcFMs) inspired by the success of generalist models, such as large language and vision models, have emerged to unify diverse human-centric tasks into a single framework, surpassing traditional task-specific approaches. In this survey, we present a comprehensive overview of HcFMs by proposing a taxonomy that categorizes current approaches into four groups: (1) Human-centric Perception Foundation Models that capture fine-grained features for multi-modal 2D and 3D understanding. (2) Human-centric AIGC Foundation Models that generate high-fidelity, diverse human-related content. (3) Unified Perception and Generation Models that integrate these capabilities to enhance both human understanding and synthesis. (4) Human-centric Agentic Foundation Models that extend beyond perception and generation to learn human-like intelligence and interactive behaviors for humanoid embodied tasks. We review state-of-the-art techniques, discuss emerging challenges and future research directions. This survey aims to serve as a roadmap for researchers and practitioners working towards more robust, versatile, and intelligent digital human and embodiments modeling.