Human-Centric Foundation Models: Perception, Generation and Agentic Modeling
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Human understanding and generation are critical for modeling digital humans and humanoid embodiments. Recently, Human-centric Foundation Models (HcFMs) inspired by the success of generalist models, such as large language and vision models, have emerged to unify diverse human-centric tasks into a single framework, surpassing traditional task-specific approaches. In this survey, we present a comprehensive overview of HcFMs by proposing a taxonomy that categorizes current approaches into four groups: (1) Human-centric Perception Foundation Models that capture fine-grained features for multi-modal 2D and 3D understanding. (2) Human-centric AIGC Foundation Models that generate high-fidelity, diverse human-related content. (3) Unified Perception and Generation Models that integrate these capabilities to enhance both human understanding and synthesis. (4) Human-centric Agentic Foundation Models that extend beyond perception and generation to learn human-like intelligence and interactive behaviors for humanoid embodied tasks. We review state-of-the-art techniques, discuss emerging challenges and future research directions. This survey aims to serve as a roadmap for researchers and practitioners working towards more robust, versatile, and intelligent digital human and embodiments modeling.
Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 50 of its 56 references that are in the map. First posted on arXiv 12 Feb 2025.
Builds on
50 of 56 references.
References · 50
- 2023GPT-4 Technical Reportcited by 18,132
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2023Visual Instruction Tuningcited by 7,792
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
- 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
- 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
- 2024Sapiens: Foundation for Human Vision Modelscited by 229
- 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
- 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 218
- 2023SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimationcited by 154
- 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
38 more
- 2022StyleGAN-Human: A Data-Centric Odyssey of Human Generationcited by 143
- 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
- 2023HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generationcited by 104
- 2023DreamHuman: Animatable 3D Avatars from Textcited by 102
- 2023Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Taskscited by 94
- 2023ChatPose: Chatting about 3D Human Posecited by 71
- 2023HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusioncited by 68
- 2024CoMo: Controllable Motion Generation through Language Guided Pose Code Editingcited by 66
- 2020SofGAN: A Portrait Image Generator with Dynamic Stylingcited by 65
- 2023AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyondcited by 63
- 2024Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformercited by 61
- 2023UniHCP: A Unified Model for Human-Centric Perceptionscited by 59
- 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understandingcited by 58
- 2023AG3D: Learning to Generate 3D Avatars from 2D Image Collectionscited by 55
- 2023Instruct-ReID: A Multi-purpose Person Re-identification Task with Instructionscited by 46
- 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
- 2023HumanBench: Towards General Human-centric Perception with Projector Assisted Pretrainingcited by 38
- 2024M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generationcited by 37
- 2023HAP: Structure-Aware Masked Image Modeling for Human-Centric Perceptioncited by 33
- 2024CosmicMan: A Text-to-Image Foundation Model for Humanscited by 30
- 2022Versatile Multi-Modal Pre-Training for Human-Centric Perceptioncited by 26
- 2023Hulk: A Universal Knowledge Translator for Human-Centric Taskscited by 22
- 2024ChatGarment: Garment Estimation, Generation and Editing via Large Language Modelscited by 22
- 2023Get3DHuman: Lifting StyleGAN-Human into a 3D Generative Model using Pixel-aligned Reconstruction Priorscited by 19
- 2024All in One Framework for Multimodal Re-identification in the Wildcited by 18
- 2024UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editingcited by 18
- 2024HumanVLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoidcited by 17
- 2023AniPortraitGAN: Animatable 3D Portrait Generation from 2D Image Collectionscited by 16
- 2024SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillationcited by 16
- 2024Fashion-VDM: Video Diffusion Model for Virtual Try-Oncited by 16
- 2024When Pedestrian Detection Meets Multi-Modal Learning: Generalist Model and Benchmark Datasetcited by 10
- 2023UnitedHuman: Harnessing Multi-Source Data for High-Resolution Human Generationcited by 9
- 2023You Only Learn One Query: Learning Unified Human Query for Single-Stage Multi-Person Multi-Task Human-Centric Perceptioncited by 9
- 2023Cross-view and Cross-pose Completion for 3D Human Understandingcited by 6
- 2024FaceGPT: Self-supervised Learning to Chat about 3D Human Facescited by 4
- 2024Bringing Robots Home: The Rise of AI Robots in Consumer Electronicscited by 2
- 2023StyleHumanCLIP: Text-guided Garment Manipulation for StyleGAN-Humancited by 1
- 2024RefHCM: A Unified Model for Referring Perceptions in Human-Centric Scenarioscited by 1
Built on by
Citing papers per year: 2025: 4; 2026: 6.
10 papers.
- 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generationcited by 17
- 2025MMGen: Unified Multi-modal Image Generation and Understanding in One Gocited by 10
- 2026VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluationcited by 2
- 2025MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generationcited by 1
- 2025FMGS-Avatar: Mesh-Guided 2D Gaussian Splatting with Foundation Model Priors for 3D Monocular Avatar Reconstructioncited by 0
- 2026The Global-Local loop: what is missing in bridging the gap between geospatial data from numerous communities?cited by 0
- 2026Marrying Text-to-Motion Generation with Skeleton-Based Action Recognitioncited by 0
- 2026Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipatescited by 0
- 2026VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Qualitycited by 0
- 2026HUG-VIS: A Multimodal Benchmark for Human-centered Understanding and Generation in Visual Intelligencecited by 0
- 2026HUG-VIS: A Multimodal Benchmark for Human-centered Understanding and Generation in Visual Intelligence27 Aug 2026
- 2026VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Quality26 Aug 2026
- 2026Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates21 Aug 2026
- 2026Marrying Text-to-Motion Generation with Skeleton-Based Action Recognition18 Apr 2026
- 2026VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation11 Apr 2026
- 2026The Global-Local loop: what is missing in bridging the gap between geospatial data from numerous communities?19 Mar 2026
- 2025FMGS-Avatar: Mesh-Guided 2D Gaussian Splatting with Foundation Model Priors for 3D Monocular Avatar Reconstruction18 Sep 2025
- 2025MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation7 Jul 2025
- 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation3 Apr 2025
- 2025MMGen: Unified Multi-modal Image Generation and Understanding in One Go26 Mar 2025
Similar, not cited
Close in wording, no citation either way
- 2026Evaluating Generative Models as Interactive Emergent Representations of Human-Like Collaborative Behavior
- 2025Foundation Model Driven Robotics: A Comprehensive Review
- 2026The Embodiment Gap in Robot Foundation Models
- 2025How Good are Foundation Models in Step-by-Step Embodied Reasoning?
- 2026Foundation Models in Robotics: A Comprehensive Review of Methods, Models, Datasets, Challenges and Future Research Directions
- 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents
- 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review
- 2026THFM: A Unified Video Foundation Model for 4D Human Perception and Beyond