# Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

arXiv: https://arxiv.org/abs/2301.12688 · HTML: https://mlsanity.com/paper/2301.12688/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 65 references that are in the map.
- First posted on arXiv 30 Jan 2023.

## Builds on (most cited first, 20 of 20)

- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples](https://mlsanity.com/paper/2103.05905/) (2021)
- [Scene-aware Generative Network for Human Motion Synthesis](https://mlsanity.com/paper/2105.14804/) (2021)
- [MOTSynth: How Can Synthetic Data Help Pedestrian Detection and Tracking?](https://mlsanity.com/paper/2108.09518/) (2021)
- [AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars](https://mlsanity.com/paper/2205.08535/) (2022)
- [A Unified Framework for Shot Type Classification Based on Subject Centric Lens](https://mlsanity.com/paper/2008.03548/) (2020)
- [MovieCuts: A New Dataset and Benchmark for Cut Type Recognition](https://mlsanity.com/paper/2109.05569/) (2021)
- [HoME: a Household Multimodal Environment](https://mlsanity.com/paper/1711.11017/) (2017)
- [Learning to Cut by Watching Movies](https://mlsanity.com/paper/2108.04294/) (2021)
- [GAZED- Gaze-guided Cinematic Editing of Wide-Angle Monocular Video Recordings](https://mlsanity.com/paper/2010.11886/) (2020)
- [Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography](https://mlsanity.com/paper/1904.02579/) (2019)
- [The Prose Storyboard Language: A Tool for Annotating and Directing Movies](https://mlsanity.com/paper/1508.07593/) (2015)
- [Directing Cinematographic Drones](https://mlsanity.com/paper/1712.04216/) (2017)

## Built on by (most cited first, 21 of 21)

- [SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and Modeling](https://mlsanity.com/paper/2303.17368/) (2023)
- [WHAC: World-grounded Humans and Cameras](https://mlsanity.com/paper/2403.12959/) (2024)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [Generative AI for Film Creation: A Survey of Recent Advances](https://mlsanity.com/paper/2504.08296/) (2025)
- [Cinematic Behavior Transfer via NeRF-based Differentiable Filming](https://mlsanity.com/paper/2311.17754/) (2023)
- [DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance](https://mlsanity.com/paper/2403.13667/) (2024)
- [CinePreGen: Camera Controllable Video Previsualization via Engine-powered Diffusion](https://mlsanity.com/paper/2408.17424/) (2024)
- [ScriptViz: A Visualization Tool to Aid Scriptwriting based on a Large Movie Database](https://mlsanity.com/paper/2410.03224/) (2024)
- [Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation](https://mlsanity.com/paper/2509.26555/) (2025)
- [Learning to Highlight Audio by Watching Movies](https://mlsanity.com/paper/2505.12154/) (2025)
- [AdaViewPlanner: Adapting Video Diffusion Models for Viewpoint Planning in 4D Scenes](https://mlsanity.com/paper/2510.10670/) (2025)
- [PrevizWhiz: Combining Rough 3D Scenes and 2D Video to Guide Generative Video Previsualization](https://mlsanity.com/paper/2602.03838/) (2026)
- [MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation](https://mlsanity.com/paper/2605.28035/) (2026)
- [DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis](https://mlsanity.com/paper/2409.14925/) (2024)
- [VERTIGO: Visual Preference Optimization for Cinematic Camera Trajectory Generation](https://mlsanity.com/paper/2604.02467/) (2026)
- [StoryBlender: Inter-Shot Consistent and Editable 3D Storyboard with Spatial-temporal Dynamics](https://mlsanity.com/paper/2604.03315/) (2026)
- [Mind-of-Director: Multi-modal Agent-Driven Film Previsualization via Collaborative Decision-Making](https://mlsanity.com/paper/2603.14790/) (2026)
- [JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents](https://mlsanity.com/paper/2607.23588/) (2026)
- [Look-Before-Move: Narrative-Grounded World Visual Attention in Dynamic 3D Story Worlds](https://mlsanity.com/paper/2606.26964/) (2026)
- [SAGE: Self-Evolving Storyboard Skills via Attribution-Guided Rule Evolution](https://mlsanity.com/paper/2608.17468/) (2026)
- [PACE: Precise AI Cinematic Expression](https://mlsanity.com/paper/2609.19853/) (2026)

## Abstract (arXiv)

Amateurs working on mini-films and short-form videos usually spend lots of time and effort on the multi-round complicated process of setting and adjusting scenes, plots, and cameras to deliver satisfying video shots. We present Virtual Dynamic Storyboard (VDS) to allow users storyboarding shots in virtual environments, where the filming staff can easily test the settings of shots before the actual filming. VDS runs on a "propose-simulate-discriminate" mode: Given a formatted story script and a camera script as input, it generates several character animation and camera movement proposals following predefined story and cinematic rules to allow an off-the-shelf simulation engine to render videos. To pick up the top-quality dynamic storyboard from the candidates, we equip it with a shot ranking discriminator based on shot quality criteria learned from professional manual-created data. VDS is comprehensively validated via extensive experiments and user studies, demonstrating its efficiency, effectiveness, and great potential in assisting amateur video production.