# SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation

arXiv: https://arxiv.org/abs/2512.01960 · HTML: https://mlsanity.com/paper/2512.01960/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 43 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Builds on (most cited first, 30 of 37)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](https://mlsanity.com/paper/2404.13026/) (2024)
- [Diffusion Adversarial Post-Training for One-Step Video Generation](https://mlsanity.com/paper/2501.08316/) (2025)
- [Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2012.09856/) (2020)
- [GameGen-X: Interactive Open-world Game Video Generation](https://mlsanity.com/paper/2411.00769/) (2024)
- [Stabilizing Training of Generative Adversarial Networks through Regularization](https://mlsanity.com/paper/1705.09367/) (2017)
- [WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generation](https://mlsanity.com/paper/2312.02934/) (2023)
- [Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation](https://mlsanity.com/paper/2506.09350/) (2025)

## Built on by (most cited first, 3 of 3)

- [Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures](https://mlsanity.com/paper/2602.09600/) (2026)
- [EgoSim: Egocentric World Simulator for Embodied Interaction Generation](https://mlsanity.com/paper/2604.01001/) (2026)
- [HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Control](https://mlsanity.com/paper/2607.02075/) (2026)

## Abstract (arXiv)

Modeling and synthesizing complex hand-object interactions remains a significant challenge, even for state-of-the-art physics engines. Conventional simulation-based approaches rely on explicitly defined rigid object models and pre-scripted hand gestures, making them inadequate for capturing dynamic interactions with non-rigid or articulated entities such as deformable fabrics, elastic materials, hinge-based structures, furry surfaces, or even living creatures. In this paper, we present SpriteHand, an autoregressive video generation framework for real-time synthesis of versatile hand-object interaction videos across a wide range of object types and motion patterns. SpriteHand takes as input a static object image and a video stream in which the hands are imagined to interact with the virtual object embedded in a real-world scene, and generates corresponding hand-object interaction effects in real time. Our model employs a causal inference architecture for autoregressive generation and leverages a hybrid post-training approach to enhance visual realism and temporal coherence. Our 1.3B model supports real-time streaming generation at around 18 FPS and 640x368 resolution, with an approximate 150 ms latency on a single NVIDIA RTX 5090 GPU, and more than a minute of continuous output. Experiments demonstrate superior visual quality, physical plausibility, and interaction fidelity compared to both generative and engine-based baselines.