mlsanityData to 14 Oct 2026

UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating

arXiv 2606.21661 ↗First posted 19 Jun 2026cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Generating a coherent multi-shot video requires structured cross-shot memory. Subject appearance, scene context, and speaker identity must persist across cuts. Existing approaches either train end-to-end over fixed-length sequences and cannot scale, generate shot-by-shot with memory banks that grow linearly, or orchestrate pretrained generators under an LLM planner without a multi-shot-aware backbone. We present UnityShots, a memory-driven multi-shot audio-video generation system built on LTX-2.3, trained on annotated cinematic and music-video shots. The video stream maintains two fixed-size slots, a long-term memory (LTM) slot anchored to the opening shot and a short-term memory (STM) slot holding the immediately preceding tail, both updated at every cut by a boundary-conditioned gate that fuses visual cut probability and beat-tracker signals. The audio stream injects a reference speaker token at every shot to preserve vocal timbre without a sliding audio bank. A discrete cut-type prior, learned through AdaLN, becomes an inference-time control knob over transition strength. We release a benchmark of 200 multi-cultural multi-shot sequences spanning six ethnic regions and ten or more languages, with per-shot reference identities, reference audio, and per-boundary transition labels. Evaluated across I2V, T2V, and R2V conditioning modes, UnityShots leads open-source baselines on every cross-shot coherence metric and matches the strongest closed-source system on the multi-shot axes.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 53 of its 56 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

53 of 56 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2025Qwen3-VL Technical Reportcited by 2,793
  9. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  11. 2022Video Diffusion Modelscited by 2,305
  12. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
38 more
  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  3. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  4. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  7. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  8. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  9. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  10. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  11. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  12. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  13. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
  14. 2020TransNet V2: An effective deep network architecture for fast shot transition detectioncited by 171
  15. 2024In-Context LoRA for Diffusion Transformerscited by 158
  16. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 143
  17. 2025Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Soundcited by 137
  18. 2025Long Context Tuning for Video Generationcited by 113
  19. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  20. 2021Hybrid Spectrogram and Waveform Source Separationcited by 91
  21. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 83
  22. 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generationcited by 74
  23. 2025HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narrativescited by 54
  24. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
  25. 2025Captain Cinema: Towards Short Movie Generationcited by 48
  26. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 43
  27. 2025StoryMem: Multi-shot Long Video Storytelling with Memorycited by 40
  28. 2025ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Modelscited by 38
  29. 2025MultiShotMaster: A Controllable Multi-Shot Video Generation Frameworkcited by 36
  30. 2025OneStory: Coherent Multi-Shot Video Generation with Adaptive Memorycited by 35
  31. 2025CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Modelscited by 29
  32. 2026ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytellingcited by 28
  33. 2024Beat this! Accurate beat tracking without DBN postprocessingcited by 27
  34. 2026VideoMemory: Toward Consistent Video Generation via Memory Integrationcited by 25
  35. 2025STAGE: Storyboard-Anchored Generation for Cinematic Multi-shot Narrativecited by 23
  36. 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 20
  37. 2025Cut2Next: Generating Next Shot via In-Context Tuningcited by 15
  38. 2025UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalistcited by 15

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.