# The Role of Video Generation in Enhancing Data-Limited Action Understanding

arXiv: https://arxiv.org/abs/2505.19495 · HTML: https://mlsanity.com/paper/2505.19495/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 70 of its 98 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 30 of 70)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)

## Built on by (most cited first, 2 of 2)

- [CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models](https://mlsanity.com/paper/2508.13446/) (2025)
- [Gen2Balance: Generative Balancing for Long-Tailed Video Action Recognition](https://mlsanity.com/paper/2606.22416/) (2026)

## Abstract (arXiv)

Video action understanding tasks in real-world scenarios always suffer data limitations. In this paper, we address the data-limited action understanding problem by bridging data scarcity. We propose a novel method that employs a text-to-video diffusion transformer to generate annotated data for model training. This paradigm enables the generation of realistic annotated data on an infinite scale without human intervention. We proposed the information enhancement strategy and the uncertainty-based label smoothing tailored to generate sample training. Through quantitative and qualitative analysis, we observed that real samples generally contain a richer level of information than generated samples. Based on this observation, the information enhancement strategy is proposed to enhance the informative content of the generated samples from two aspects: the environments and the characters. Furthermore, we observed that some low-quality generated samples might negatively affect model training. To address this, we devised the uncertainty-based label smoothing strategy to increase the smoothing of these samples, thus reducing their impact. We demonstrate the effectiveness of the proposed method on four datasets across five tasks and achieve state-of-the-art performance for zero-shot action recognition.