# OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulation

arXiv: https://arxiv.org/abs/2508.19209 · HTML: https://mlsanity.com/paper/2508.19209/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 82 of its 94 references that are in the map.
- First posted on arXiv 26 Aug 2025.

## Builds on (most cited first, 30 of 82)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)

## Built on by (most cited first, 30 of 32)

- [Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesis](https://mlsanity.com/paper/2509.09595/) (2025)
- [AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement](https://mlsanity.com/paper/2511.23475/) (2025)
- [KlingAvatar 2.0 Technical Report](https://mlsanity.com/paper/2512.13313/) (2025)
- [SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model](https://mlsanity.com/paper/2602.21818/) (2026)
- [Grounding World Simulation Models in a Real-World Metropolis](https://mlsanity.com/paper/2603.15583/) (2026)
- [Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation](https://mlsanity.com/paper/2512.21734/) (2025)
- [FlowAct-R1: Towards Interactive Humanoid Video Generation](https://mlsanity.com/paper/2601.10103/) (2026)
- [LPM 1.0: Video-based Character Performance Model](https://mlsanity.com/paper/2604.07823/) (2026)
- [ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body](https://mlsanity.com/paper/2512.14234/) (2025)
- [AvatarForcing: One-Step Streaming Talking Avatars via Local-Future Sliding-Window Denoising](https://mlsanity.com/paper/2603.14331/) (2026)
- [LongCat-Video-Avatar 1.5 Technical Report](https://mlsanity.com/paper/2605.26486/) (2026)
- [ActAvatar: Temporally-Aware Precise Action Control for Talking Avatars](https://mlsanity.com/paper/2512.19546/) (2025)
- [Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation](https://mlsanity.com/paper/2512.13495/) (2025)
- [TalkVerse: Democratizing Minute-Long Audio-Driven Video Generation](https://mlsanity.com/paper/2512.14938/) (2025)
- [Active Intelligence in Video Avatars via Closed-loop World Modeling](https://mlsanity.com/paper/2512.20615/) (2025)
- [Gloria: Consistent Character Video Generation via Content Anchors](https://mlsanity.com/paper/2603.29931/) (2026)
- [StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration](https://mlsanity.com/paper/2605.25659/) (2026)
- [EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers](https://mlsanity.com/paper/2601.22127/) (2026)
- [Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model](https://mlsanity.com/paper/2608.05976/) (2026)
- [JoyStreamer: Unlocking Highly Expressive Avatars via Harmonized Text-Audio Conditioning](https://mlsanity.com/paper/2602.00702/) (2026)
- [AUHead: Realistic Emotional Talking Head Generation via Action Units Control](https://mlsanity.com/paper/2602.09534/) (2026)
- [InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidance](https://mlsanity.com/paper/2603.23132/) (2026)
- [Avatar V: Scaling Video-Reference Avatar Video Generation](https://mlsanity.com/paper/2606.13872/) (2026)
- [Wan-Dancer: A Hierarchical Framework for Minute-scale Coherent Music-to-Dance Generation](https://mlsanity.com/paper/2607.09581/) (2026)
- [AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment](https://mlsanity.com/paper/2607.22241/) (2026)
- [YingVideo-MV: Music-Driven Multi-Stage Video Generation](https://mlsanity.com/paper/2512.02492/) (2025)
- [Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understanding](https://mlsanity.com/paper/2601.07298/) (2026)
- [EchoTorrent: Towards Swift, Sustained, and Streaming Multi-Modal Video Generation](https://mlsanity.com/paper/2602.13669/) (2026)
- [ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generation](https://mlsanity.com/paper/2603.17427/) (2026)
- [Human-Centric Intelligence in the Era of Foundation Models: A Survey](https://mlsanity.com/paper/2608.18184/) (2026)

## Abstract (arXiv)

Existing video avatar models can produce fluid human animations, yet they struggle to move beyond mere physical likeness to capture a character's authentic essence. Their motions typically synchronize with low-level cues like audio rhythm, lacking a deeper semantic understanding of emotion, intent, or context. To bridge this gap, we propose a framework designed to generate character animations that are not only physically plausible but also semantically coherent and expressive. Our model, OmniHuman-1.5, is built upon two key technical contributions. First, we leverage Multimodal Large Language Models to synthesize a structured textual representation of conditions that provides high-level semantic guidance. This guidance steers our motion generator beyond simplistic rhythmic synchronization, enabling the production of actions that are contextually and emotionally resonant. Second, to ensure the effective fusion of these multimodal inputs and mitigate inter-modality conflicts, we introduce a specialized Multimodal DiT architecture with a novel Pseudo Last Frame design. The synergy of these components allows our model to accurately interpret the joint semantics of audio, images, and text, thereby generating motions that are deeply coherent with the character, scene, and linguistic content. Extensive experiments demonstrate that our model achieves leading performance across a comprehensive set of metrics, including lip-sync accuracy, video quality, motion naturalness and semantic consistency with textual prompts. Furthermore, our approach shows remarkable extensibility to complex scenarios, such as those involving multi-person and non-human subjects. Homepage: hrefhttps://omnihuman-lab.github.io/v15/