# FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios

arXiv: https://arxiv.org/abs/2505.03730 · HTML: https://mlsanity.com/paper/2505.03730/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 6 May 2025.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://mlsanity.com/paper/2305.08891/) (2023)
- [Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2309.15818/) (2023)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance](https://mlsanity.com/paper/2406.19680/) (2024)
- [FreeU: Free Lunch in Diffusion U-Net](https://mlsanity.com/paper/2309.11497/) (2023)
- [ControlNeXt: Powerful and Efficient Control for Image and Video Generation](https://mlsanity.com/paper/2408.06070/) (2024)
- [Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation](https://mlsanity.com/paper/2305.10874/) (2023)
- [MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions](https://mlsanity.com/paper/2407.06358/) (2024)
- [DisCo: Disentangled Control for Realistic Human Dance Generation](https://mlsanity.com/paper/2307.00040/) (2023)
- [Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer](https://mlsanity.com/paper/2311.17009/) (2023)
- [MotionClone: Training-Free Motion Cloning for Controllable Video Generation](https://mlsanity.com/paper/2406.05338/) (2024)
- [FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention](https://mlsanity.com/paper/2407.19918/) (2024)
- [StableAnimator: High-Quality Identity-Preserving Human Image Animation](https://mlsanity.com/paper/2411.17697/) (2024)
- [InstructVideo: Instructing Video Diffusion Models with Human Feedback](https://mlsanity.com/paper/2312.12490/) (2023)
- [Allegro: Open the Black Box of Commercial-Level Video Generation Model](https://mlsanity.com/paper/2410.15458/) (2024)

## Built on by (most cited first, 25 of 25)

- [Video-As-Prompt: Unified Semantic Control for Video Generation](https://mlsanity.com/paper/2510.20888/) (2025)
- [VFXMaster: Unlocking Dynamic Visual Effect Generation via In-Context Learning](https://mlsanity.com/paper/2510.25772/) (2025)
- [Frame In-N-Out: Unbounded Controllable Image-to-Video Generation](https://mlsanity.com/paper/2505.21491/) (2025)
- [Animate-X++: Universal Character Image Animation with Dynamic Backgrounds](https://mlsanity.com/paper/2508.09454/) (2025)
- [O-DisCo-Edit: Object Distortion Control for Unified Realistic Video Editing](https://mlsanity.com/paper/2509.01596/) (2025)
- [SteadyDancer: Harmonized and Coherent Human Image Animation with First-Frame Preservation](https://mlsanity.com/paper/2511.19320/) (2025)
- [SMRABooth: Subject and Motion Representation Alignment for Customized Video Generation](https://mlsanity.com/paper/2512.12193/) (2025)
- [SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation](https://mlsanity.com/paper/2506.23690/) (2025)
- [OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer](https://mlsanity.com/paper/2601.14250/) (2026)
- [DreamActor-M2: Universal Character Image Animation via Spatiotemporal In-Context Learning](https://mlsanity.com/paper/2601.21716/) (2026)
- [TS-Attn: Temporal-wise Separable Attention for Multi-Event Video Generation](https://mlsanity.com/paper/2604.19473/) (2026)
- [From Inpainting to Editing: Unlocking Robust Mask-Free Visual Dubbing via Generative Bootstrapping](https://mlsanity.com/paper/2512.25066/) (2025)
- [Let Your Image Move with Your Motion! -- Implicit Multi-Object Multi-Motion Transfer](https://mlsanity.com/paper/2603.01000/) (2026)
- [Meta-CoT: Enhancing Granularity and Generalization in Image Editing](https://mlsanity.com/paper/2604.24625/) (2026)
- [ConceptWeaver: Weaving Disentangled Concepts with Flow](https://mlsanity.com/paper/2603.28493/) (2026)
- [EasyVFX: Frequency-Driven Decoupling for Resource-Efficient VFX Generation](https://mlsanity.com/paper/2605.22051/) (2026)
- [Characterizing Motion Encoding in Video Diffusion Timesteps](https://mlsanity.com/paper/2512.22175/) (2025)
- [IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animation](https://mlsanity.com/paper/2602.07498/) (2026)
- [Few-Shot-Based Modular Image-to-Video Adapter for Diffusion Models](https://mlsanity.com/paper/2512.20000/) (2025)
- [Screen, Cache, and Match: A Training-Free Causality-Consistent Reference Frame Framework for Human Animation](https://mlsanity.com/paper/2601.22160/) (2025)
- [Video2LoRA: Unified Semantic-Controlled Video Generation via Per-Reference-Video LoRA](https://mlsanity.com/paper/2603.08210/) (2026)
- [AnyCrowd: Instance-Isolated Identity-Pose Binding for Arbitrary Multi-Character Animation](https://mlsanity.com/paper/2603.15415/) (2026)
- [Disco-LoRA: Disentangled Composition of Content, Style, and Motion for Multi-concept Video Customization](https://mlsanity.com/paper/2606.26668/) (2026)
- [EMOSH: Expressive Motion and Shape Disentanglement for Human Animation](https://mlsanity.com/paper/2606.28026/) (2026)
- [Motion4Motion: Motion Transfer Across Subjects at Inference](https://mlsanity.com/paper/2607.11644/) (2026)

## Abstract (arXiv)

Action customization involves generating videos where the subject performs actions dictated by input control signals. Current methods use pose-guided or global motion customization but are limited by strict constraints on spatial structure, such as layout, skeleton, and viewpoint consistency, reducing adaptability across diverse subjects and scenarios. To overcome these limitations, we propose FlexiAct, which transfers actions from a reference video to an arbitrary target image. Unlike existing methods, FlexiAct allows for variations in layout, viewpoint, and skeletal structure between the subject of the reference video and the target image, while maintaining identity consistency. Achieving this requires precise action control, spatial structure adaptation, and consistency preservation. To this end, we introduce RefAdapter, a lightweight image-conditioned adapter that excels in spatial adaptation and consistency preservation, surpassing existing methods in balancing appearance consistency and structural flexibility. Additionally, based on our observations, the denoising process exhibits varying levels of attention to motion (low frequency) and appearance details (high frequency) at different timesteps. So we propose FAE (Frequency-aware Action Extraction), which, unlike existing methods that rely on separate spatial-temporal architectures, directly achieves action extraction during the denoising process. Experiments demonstrate that our method effectively transfers actions to subjects with diverse layouts, skeletons, and viewpoints. We release our code and model weights to support further research at https://shiyi-zh0408.github.io/projectpages/FlexiAct/