# Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation

arXiv: https://arxiv.org/abs/2501.05020 · HTML: https://mlsanity.com/paper/2501.05020/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 51 references that are in the map.
- First posted on arXiv 9 Jan 2025.

## Builds on (most cited first, 30 of 45)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis](https://mlsanity.com/paper/2409.02048/) (2024)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos](https://mlsanity.com/paper/2304.01186/) (2023)
- [DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory](https://mlsanity.com/paper/2308.08089/) (2023)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [SpatialTracker: Tracking Any 2D Pixels in 3D Space](https://mlsanity.com/paper/2404.04319/) (2024)
- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://mlsanity.com/paper/2311.16933/) (2023)
- [CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation](https://mlsanity.com/paper/2406.02509/) (2024)
- [Tora: Trajectory-oriented Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2407.21705/) (2024)
- [DragAnything: Motion Control for Anything using Entity Representation](https://mlsanity.com/paper/2403.07420/) (2024)

## Built on by (most cited first, 23 of 23)

- [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025)
- [Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation](https://mlsanity.com/paper/2504.14899/) (2025)
- [CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation](https://mlsanity.com/paper/2502.08639/) (2025)
- [Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance](https://mlsanity.com/paper/2512.08765/) (2025)
- [ATI: Any Trajectory Instruction for Controllable Video Generation](https://mlsanity.com/paper/2505.22944/) (2025)
- [VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control](https://mlsanity.com/paper/2601.05138/) (2026)
- [VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation](https://mlsanity.com/paper/2502.07531/) (2025)
- [Generative Video Motion Editing with 3D Point Tracks](https://mlsanity.com/paper/2512.02015/) (2025)
- [MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agent](https://mlsanity.com/paper/2502.03207/) (2025)
- [SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generation](https://mlsanity.com/paper/2604.03723/) (2026)
- [McSc: Motion-Corrective Preference Alignment for Video Generation with Self-Critic Hierarchical Reasoning](https://mlsanity.com/paper/2511.22974/) (2025)
- [ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer](https://mlsanity.com/paper/2603.15478/) (2026)
- [MoRight: Motion Control Done Right](https://mlsanity.com/paper/2604.07348/) (2026)
- [OptiWorld: Optimal Control for Video World Generation under Physical Constraints](https://mlsanity.com/paper/2606.00499/) (2026)
- [FlowLoss: Dynamic Flow-Conditioned Loss Strategy for Video Diffusion Models](https://mlsanity.com/paper/2504.14535/) (2025)
- [Preserve, Reveal, Expand: Faithful 4D Video Editing with Region-Aware Conditioning](https://mlsanity.com/paper/2605.20961/) (2026)
- [Physical Object Understanding with a Physically Controllable World Model](https://mlsanity.com/paper/2606.00439/) (2026)
- [Towards 3D-Aware Video Diffusion Models: Render-Free Human Motion Control with Mesh Tokenization](https://mlsanity.com/paper/2606.02000/) (2026)
- [Geometry-Instructed Video Editing](https://mlsanity.com/paper/2606.24225/) (2026)
- [CoaG: Cylinders on a Grid for Coarse 3D Layout Control in Video Generation](https://mlsanity.com/paper/2609.24208/) (2026)
- [LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation](https://mlsanity.com/paper/2609.38146/) (2026)
- [4Director: Controlling Video World Models with Rigid 3D Geometry](https://mlsanity.com/paper/2610.02160/) (2026)
- [Gen Through the Lens: Camera-Controllable Video Generation via Cross-Coordinate Conditioning Fusion](https://mlsanity.com/paper/2610.15900/) (2026)

## Abstract (arXiv)

Motion-controllable image animation is a fundamental task with a wide range of potential applications. Recent works have made progress in controlling camera or object motion via various motion representations, while they still struggle to support collaborative camera and object motion control with adaptive control granularity. To this end, we introduce 3D-aware motion representation and propose an image animation framework, called Perception-as-Control, to achieve fine-grained collaborative motion control. Specifically, we construct 3D-aware motion representation from a reference image, manipulate it based on interpreted user instructions, and perceive it from different viewpoints. In this way, camera and object motions are transformed into intuitive and consistent visual changes. Then, our framework leverages the perception results as motion control signals, enabling it to support various motion-related video synthesis tasks in a unified and flexible way. Experiments demonstrate the superiority of the proposed approach. For more details and qualitative results, please refer to our anonymous project webpage: https://chen-yingjie.github.io/projects/Perception-as-Control.