# The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation

arXiv: https://arxiv.org/abs/2609.02367 · HTML: https://mlsanity.com/paper/2609.02367/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 48 of its 56 references that are in the map.
- First posted on arXiv 2 Sep 2026.

## Builds on (most cited first, 30 of 48)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [LTX-2: Efficient Joint Audio-Visual Foundation Model](https://mlsanity.com/paper/2601.03233/) (2026)
- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization](https://mlsanity.com/paper/2503.23377/) (2025)
- [Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation](https://mlsanity.com/paper/2510.01284/) (2025)
- [Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation](https://mlsanity.com/paper/2309.16429/) (2023)
- [HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narratives](https://mlsanity.com/paper/2510.20822/) (2025)
- [Modeling Localness for Self-Attention Networks](https://mlsanity.com/paper/1810.10182/) (2018)
- [AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation](https://mlsanity.com/paper/2406.07686/) (2024)
- [Mind the Time: Temporally-Controlled Multi-Event Video Generation](https://mlsanity.com/paper/2412.05263/) (2024)
- [ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models](https://mlsanity.com/paper/2505.07652/) (2025)
- [UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions](https://mlsanity.com/paper/2511.03334/) (2025)
- [MultiShotMaster: A Controllable Multi-Shot Video Generation Framework](https://mlsanity.com/paper/2512.03041/) (2025)
- [CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Models](https://mlsanity.com/paper/2508.11484/) (2025)
- [JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation](https://mlsanity.com/paper/2602.19163/) (2026)
- [AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation](https://mlsanity.com/paper/2412.15191/) (2024)
- [Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy](https://mlsanity.com/paper/2511.21579/) (2025)
- [A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation](https://mlsanity.com/paper/2409.17550/) (2024)
- [Audio-Sync Video Generation with Multi-Stream Temporal Control](https://mlsanity.com/paper/2506.08003/) (2025)
- [RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time](https://mlsanity.com/paper/2604.11626/) (2026)
- [Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation](https://mlsanity.com/paper/2604.23632/) (2026)

## Abstract (arXiv)

Joint audio-video generation models have made substantial progress in visual quality and audio-visual synchronization. However, they still provide limited control over when shot transitions occur and dialogue is spoken. This limitation constrains their application in script-driven content creation, where timing errors can undermine narrative coherence and the viewing experience. Current joint generators align video and audio representations on a shared temporal axis, yet the precise timing of shots and dialogue specified in a structured prompt is encoded only in the prompt's text representation and remains unaligned with the temporal coordinates of either modality. Consequently, video and audio may remain synchronized with each other while both fail to follow the script timeline. This mismatch motivates us to extend temporal alignment beyond video and audio to include the structured script. We therefore introduce Temporal Context Routing (TCR), which maps the script timing onto the shared temporal axis of video and audio generation and routes each prompt's guidance to the corresponding positions in both modalities. Compared with the baseline on 200 test scripts, TCR reduces Shot Boundary MAE by 96%, from 1.11 s to 0.042 s, and raises Dialogue Acc@0.5 s from 28.3% to 84.1%. TCR achieves these improvements while maintaining visual quality and audio-visual synchronization comparable to those of the baselines. A user study further shows that participants prefer TCR on all five evaluated dimensions.