# DragEntity: Trajectory Guided Video Generation using Entity and Positional Relationships

arXiv: https://arxiv.org/abs/2410.10751 · HTML: https://mlsanity.com/paper/2410.10751/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023)
- [GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions](https://mlsanity.com/paper/2104.14806/) (2021)
- [N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion](https://mlsanity.com/paper/2111.12417/) (2021)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation](https://mlsanity.com/paper/2303.12346/) (2023)
- [DragAnything: Motion Control for Anything using Entity Representation](https://mlsanity.com/paper/2403.07420/) (2024)
- [Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing](https://mlsanity.com/paper/1703.05446/) (2017)
- [TrailBlazer: Trajectory Control for Diffusion-Based Video Generation](https://mlsanity.com/paper/2401.00896/) (2023)
- [VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence](https://mlsanity.com/paper/2312.02087/) (2023)
- [Recurrent Environment Simulators](https://mlsanity.com/paper/1704.02254/) (2017)
- [iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis](https://mlsanity.com/paper/2107.02790/) (2021)
- [Click to Move: Controlling Video Generation with Sparse Motion](https://mlsanity.com/paper/2108.08815/) (2021)
- [ParNet: Position-aware Aggregated Relation Network for Image-Text matching](https://mlsanity.com/paper/1906.06892/) (2019)

## Built on by (most cited first, 9 of 9)

- [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025)
- [SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation](https://mlsanity.com/paper/2411.04989/) (2024)
- [VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control](https://mlsanity.com/paper/2601.05138/) (2026)
- [VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation](https://mlsanity.com/paper/2502.07531/) (2025)
- [X-UniMotion: Animating Human Images with Expressive, Unified and Identity-Agnostic Motion Latents](https://mlsanity.com/paper/2508.09383/) (2025)
- [C-Drag: Chain-of-Thought Driven Motion Controller for Video Generation](https://mlsanity.com/paper/2502.19868/) (2025)
- [Learning to Generate Rigid Body Interactions with Video Diffusion Models](https://mlsanity.com/paper/2510.02284/) (2025)
- [TrajLoc: Trajectory-Attention Localization for Multi-Object Motion Control](https://mlsanity.com/paper/2607.00861/) (2026)
- [Generative Cinematographer: Composing Camera and Object Motion in 3D](https://mlsanity.com/paper/2610.02180/) (2026)

## Abstract (arXiv)

In recent years, diffusion models have achieved tremendous success in the field of video generation, with controllable video generation receiving significant attention. However, existing control methods still face two limitations: Firstly, control conditions (such as depth maps, 3D Mesh) are difficult for ordinary users to obtain directly. Secondly, it's challenging to drive multiple objects through complex motions with multiple trajectories simultaneously. In this paper, we introduce DragEntity, a video generation model that utilizes entity representation for controlling the motion of multiple objects. Compared to previous methods, DragEntity offers two main advantages: 1) Our method is more user-friendly for interaction because it allows users to drag entities within the image rather than individual pixels. 2) We use entity representation to represent any object in the image, and multiple objects can maintain relative spatial relationships. Therefore, we allow multiple trajectories to control multiple objects in the image with different levels of complexity simultaneously. Our experiments validate the effectiveness of DragEntity, demonstrating its excellent performance in fine-grained control in video generation.