# Digital Life Project: Autonomous 3D Characters with Social Intelligence

arXiv: https://arxiv.org/abs/2312.04547 · HTML: https://mlsanity.com/paper/2312.04547/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 109 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 30 of 58)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)

## Built on by (most cited first, 30 of 43)

- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning](https://mlsanity.com/paper/2410.07093/) (2024)
- [InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling](https://mlsanity.com/paper/2410.10010/) (2024)
- [WHAC: World-grounded Humans and Cameras](https://mlsanity.com/paper/2403.12959/) (2024)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [in2IN: Leveraging individual Information to Generate Human INteractions](https://mlsanity.com/paper/2404.09988/) (2024)
- [Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation](https://mlsanity.com/paper/2502.20370/) (2025)
- [TIMotion: Temporal and Interactive Framework for Efficient Human-Human Motion Generation](https://mlsanity.com/paper/2408.17135/) (2024)
- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis](https://mlsanity.com/paper/2508.02106/) (2025)
- [Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editing](https://mlsanity.com/paper/2410.18977/) (2024)
- [SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characters](https://mlsanity.com/paper/2412.00174/) (2024)
- [Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives](https://mlsanity.com/paper/2402.11051/) (2024)
- [Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes](https://mlsanity.com/paper/2410.10790/) (2024)
- [SocialGen: Modeling Multi-Human Social Interaction with Language Models](https://mlsanity.com/paper/2503.22906/) (2025)
- [MixerMDM: Learnable Composition of Human Motion Diffusion Models](https://mlsanity.com/paper/2504.01019/) (2025)
- [RayFlow: Instance-Aware Diffusion Acceleration via Adaptive Flow Trajectories](https://mlsanity.com/paper/2503.07699/) (2025)
- [Text2Interact: High-Fidelity and Diverse Text-to-Two-Person Interaction Generation](https://mlsanity.com/paper/2510.06504/) (2025)
- [Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Models](https://mlsanity.com/paper/2408.11801/) (2024)
- [StickMotion: Generating 3D Human Motions by Drawing a Stickman](https://mlsanity.com/paper/2503.04829/) (2025)
- [MIBURI: Towards Expressive Interactive Gesture Synthesis](https://mlsanity.com/paper/2603.03282/) (2026)
- [ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body](https://mlsanity.com/paper/2512.14234/) (2025)
- [Interact2Ar: Full-Body Human-Human Interaction Generation via Autoregressive Diffusion Models](https://mlsanity.com/paper/2512.19692/) (2025)
- [MotionVerse: A Unified Multimodal Framework for Motion Comprehension, Generation and Editing](https://mlsanity.com/paper/2509.23635/) (2025)
- [InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphs](https://mlsanity.com/paper/2512.07410/) (2025)
- [CrowdMoGen: Zero-Shot Text-Driven Collective Motion Generation](https://mlsanity.com/paper/2407.06188/) (2024)
- [PhysiInter: Integrating Physical Mapping for High-Fidelity Human Interaction Generation](https://mlsanity.com/paper/2506.07456/) (2025)
- [Event-Driven Storytelling with Multiple Lifelike Humans in a 3D Scene](https://mlsanity.com/paper/2507.19232/) (2025)
- [HINT: Hierarchical Interaction Modeling for Autoregressive Multi-Human Motion Generation](https://mlsanity.com/paper/2601.20383/) (2026)
- [Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation](https://mlsanity.com/paper/2603.00144/) (2026)

## Abstract (arXiv)

In this work, we present Digital Life Project, a framework utilizing language as the universal medium to build autonomous 3D characters, who are capable of engaging in social interactions and expressing with articulated body motions, thereby simulating life in a digital environment. Our framework comprises two primary components: 1) SocioMind: a meticulously crafted digital brain that models personalities with systematic few-shot exemplars, incorporates a reflection process based on psychology principles, and emulates autonomy by initiating dialogue topics; 2) MoMat-MoGen: a text-driven motion synthesis paradigm for controlling the character's digital body. It integrates motion matching, a proven industry technique to ensure motion quality, with cutting-edge advancements in motion generation for diversity. Extensive experiments demonstrate that each module achieves state-of-the-art performance in its respective domain. Collectively, they enable virtual characters to initiate and sustain dialogues autonomously, while evolving their socio-psychological states. Concurrently, these characters can perform contextually relevant bodily movements. Additionally, a motion captioning module further allows the virtual character to recognize and appropriately respond to human players' actions. Homepage: https://digital-life-project.com/