# LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model

arXiv: https://arxiv.org/abs/2603.27449 · HTML: https://mlsanity.com/paper/2603.27449/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 51 of its 79 references that are in the map.
- First posted on arXiv 28 Mar 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 30 of 51)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Video models are zero-shot learners and reasoners](https://mlsanity.com/paper/2509.20328/) (2025)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023)
- [How Far is Video Generation from World Model: A Physical Law Perspective](https://mlsanity.com/paper/2411.02385/) (2024)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [Matrix-game 2.0: An open-source real-time and streaming interactive world model](https://mlsanity.com/paper/2508.13009/) (2025)
- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation](https://mlsanity.com/paper/2410.05363/) (2024)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild](https://mlsanity.com/paper/2004.01946/) (2020)
- [Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2012.09856/) (2020)
- [GameGen-X: Interactive Open-world Game Video Generation](https://mlsanity.com/paper/2411.00769/) (2024)
- [Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise](https://mlsanity.com/paper/2501.08331/) (2025)
- [Affordance Diffusion: Synthesizing Hand-Object Interactions](https://mlsanity.com/paper/2303.12538/) (2023)
- [Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition](https://mlsanity.com/paper/2506.17201/) (2025)

## Built on by (most cited first, 13 of 13)

- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers](https://mlsanity.com/paper/2607.21594/) (2026)
- [PhysAgent: Automating Physics-Based 4D Synthesis via Trajectory-Grounded Multi-Agent Feedback](https://mlsanity.com/paper/2606.08688/) (2026)
- [Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control](https://mlsanity.com/paper/2607.03964/) (2026)
- [EgoExo-WM: Unlocking Exo Video for Ego World Models](https://mlsanity.com/paper/2605.15477/) (2026)
- [AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization](https://mlsanity.com/paper/2606.07326/) (2026)
- [RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation](https://mlsanity.com/paper/2607.06558/) (2026)
- [Real-Time Human-Centric World Modeling for Upper-Body Human-Object Interaction](https://mlsanity.com/paper/2607.23517/) (2026)
- [EgoPlay: Event-Triggered Video Editing for Egocentric Streams](https://mlsanity.com/paper/2607.24560/) (2026)
- [IMPACT: Attention Is the Interaction Map for Scalable Interaction-Aware World Model Training](https://mlsanity.com/paper/2609.00161/) (2026)
- [Unified Video-Action Joint Denoising for Dexterous Action and Data Generation](https://mlsanity.com/paper/2606.03868/) (2026)
- [Human-Centric Intelligence in the Era of Foundation Models: A Survey](https://mlsanity.com/paper/2608.18184/) (2026)
- [Code Plans, Diffusion Renders: Open-Ended Generative World Modeling](https://mlsanity.com/paper/2609.26458/) (2026)

## Abstract (arXiv)

Learning human-object manipulation presents significant challenges due to its fine-grained and contact-rich nature of the motions involved. Traditional physics-based animation requires extensive modeling and manual setup, and more importantly, it neither generalizes well across diverse object morphologies nor scales effectively to real-world environment. To address these limitations, we introduce LOME, an egocentric world model that can generate realistic human-object interactions as videos conditioned on an input image, a text prompt, and per-frame human actions, including both body poses and hand gestures. LOME injects strong and precise action guidance into object manipulation by jointly estimating spatial human actions and the environment contexts during training. After finetuning a pretrained video generative model on videos of diverse egocentric human-object interactions, LOME demonstrates not only high action-following accuracy and strong generalization to unseen scenarios, but also realistic physical consequences of hand-object interactions, e.g., liquid flowing from a bottle into a mug after executing a ``pouring'' action. Extensive experiments demonstrate that our video-based framework significantly outperforms state-of-the-art image based and video-based action-conditioned methods and Image/Text-to-Video (I/T2V) generative model in terms of both temporal consistency and motion control. LOME paves the way for photorealistic AR/VR experiences and scalable robotic training, without being limited to simulated environments or relying on explicit 3D/4D modeling.