UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating
Not on the map. Why?
Abstract (arXiv)
Generating a coherent multi-shot video requires structured cross-shot memory. Subject appearance, scene context, and speaker identity must persist across cuts. Existing approaches either train end-to-end over fixed-length sequences and cannot scale, generate shot-by-shot with memory banks that grow linearly, or orchestrate pretrained generators under an LLM planner without a multi-shot-aware backbone. We present UnityShots, a memory-driven multi-shot audio-video generation system built on LTX-2.3, trained on annotated cinematic and music-video shots. The video stream maintains two fixed-size slots, a long-term memory (LTM) slot anchored to the opening shot and a short-term memory (STM) slot holding the immediately preceding tail, both updated at every cut by a boundary-conditioned gate that fuses visual cut probability and beat-tracker signals. The audio stream injects a reference speaker token at every shot to preserve vocal timbre without a sliding audio bank. A discrete cut-type prior, learned through AdaLN, becomes an inference-time control knob over transition strength. We release a benchmark of 200 multi-cultural multi-shot sequences spanning six ethnic regions and ten or more languages, with per-shot reference identities, reference audio, and per-boundary transition labels. Evaluated across I2V, T2V, and R2V conditioning modes, UnityShots leads open-source baselines on every cross-shot coherence metric and matches the strongest closed-source system on the multi-shot axes.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 53 of its 56 references that are in the map. First posted on arXiv 19 Jun 2026.
Builds on
53 of 56 references.
References · 53
- 2017Attention Is All You Needcited by 60,361
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2020Denoising Diffusion Implicit Modelscited by 9,781
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
- 2022Video Diffusion Modelscited by 2,305
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
38 more
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
- 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
- 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
- 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
- 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
- 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
- 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
- 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
- 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
- 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
- 2020TransNet V2: An effective deep network architecture for fast shot transition detectioncited by 171
- 2024In-Context LoRA for Diffusion Transformerscited by 158
- 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 143
- 2025Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Soundcited by 137
- 2025Long Context Tuning for Video Generationcited by 113
- 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
- 2021Hybrid Spectrogram and Waveform Source Separationcited by 91
- 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 83
- 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generationcited by 74
- 2025HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narrativescited by 54
- 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
- 2025Captain Cinema: Towards Short Movie Generationcited by 48
- 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 43
- 2025StoryMem: Multi-shot Long Video Storytelling with Memorycited by 40
- 2025ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Modelscited by 38
- 2025MultiShotMaster: A Controllable Multi-Shot Video Generation Frameworkcited by 36
- 2025OneStory: Coherent Multi-Shot Video Generation with Adaptive Memorycited by 35
- 2025CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Modelscited by 29
- 2026ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytellingcited by 28
- 2024Beat this! Accurate beat tracking without DBN postprocessingcited by 27
- 2026VideoMemory: Toward Consistent Video Generation via Memory Integrationcited by 25
- 2025STAGE: Storyboard-Anchored Generation for Cinematic Multi-shot Narrativecited by 23
- 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 20
- 2025Cut2Next: Generating Next Shot via In-Context Tuningcited by 15
- 2025UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalistcited by 15
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Vorch-Director: Interactive World Story Model via Noise-Aware Error Rectificationcited by 1
- 2026ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routingcited by 0
- 2026PertMind: Eliciting Emergent Biological Reasoning in LLM via Reinforcement Learning on Cellular Perturbation Datacited by 0
- 2026DramaChain Bench: An End-to-End Benchmark for Short-Drama Generationcited by 0
- 2026Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generationcited by 0
- 2026REEL: Learning to Retrieve, Evaluate, and Evolve Visual Memory for Any Long-Horizon Multi-Shot Video Generationcited by 0
- 2026REEL: Learning to Retrieve, Evaluate, and Evolve Visual Memory for Any Long-Horizon Multi-Shot Video Generation12 Oct 2026
- 2026Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generation11 Oct 2026
- 2026DramaChain Bench: An End-to-End Benchmark for Short-Drama Generation1 Sep 2026
- 2026PertMind: Eliciting Emergent Biological Reasoning in LLM via Reinforcement Learning on Cellular Perturbation Data17 Aug 2026
- 2026Vorch-Director: Interactive World Story Model via Noise-Aware Error Rectification6 Aug 2026
- 2026ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routing5 Aug 2026