# TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models

arXiv: https://arxiv.org/abs/2609.01277 · HTML: https://mlsanity.com/paper/2609.01277/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 39 of its 45 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 39)

- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [LTX-2: Efficient Joint Audio-Visual Foundation Model](https://mlsanity.com/paper/2601.03233/) (2026)
- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound](https://mlsanity.com/paper/2502.05139/) (2025)
- [DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation](https://mlsanity.com/paper/2301.03786/) (2023)
- [MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis](https://mlsanity.com/paper/2412.15322/) (2024)
- [Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners](https://mlsanity.com/paper/2402.17723/) (2024)
- [Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation](https://mlsanity.com/paper/2410.07718/) (2024)
- [Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation](https://mlsanity.com/paper/2305.18474/) (2023)
- [JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization](https://mlsanity.com/paper/2503.23377/) (2025)
- [SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis](https://mlsanity.com/paper/2311.17590/) (2023)
- [Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation](https://mlsanity.com/paper/2510.01284/) (2025)
- [Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation](https://mlsanity.com/paper/2309.16429/) (2023)
- [Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors](https://mlsanity.com/paper/2212.04248/) (2022)
- [VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis](https://mlsanity.com/paper/2403.08764/) (2024)
- [Video-Guided Foley Sound Generation with Multimodal Controls](https://mlsanity.com/paper/2411.17698/) (2024)
- [MOVA: Towards Scalable and Synchronized Video-Audio Generation](https://mlsanity.com/paper/2602.08794/) (2026)
- [Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model](https://mlsanity.com/paper/2603.21986/) (2026)
- [FreeAudio: Training-Free Timing Planning for Controllable Long-Form Text-to-Audio Generation](https://mlsanity.com/paper/2507.08557/) (2025)
- [Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy](https://mlsanity.com/paper/2511.21579/) (2025)
- [AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation](https://mlsanity.com/paper/2604.08540/) (2026)
- [AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion](https://mlsanity.com/paper/2305.04001/) (2023)
- [VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation](https://mlsanity.com/paper/2412.10768/) (2024)
- [Audio-Sync Video Generation with Multi-Stream Temporal Control](https://mlsanity.com/paper/2506.08003/) (2025)
- [PicoAudio: Enabling Precise Timestamp and Frequency Controllability of Audio Events in Text-to-audio Generation](https://mlsanity.com/paper/2407.02869/) (2024)
- [ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling](https://mlsanity.com/paper/2510.08878/) (2025)

## Abstract (arXiv)

Although pretrained joint audio-visual diffusion models offer rich control over what to generate, they provide no explicit control over when an utterance should occur. To address this, we study inference-time speech scheduling, a novel task that places coupled speech and visual articulation within user-specified begin--end intervals without finetuning the backbone model. We uncover two intrinsic properties of the denoising process that enable this task. First, a timing-sensitive text-to-audio cross-attention head exposes each utterance's model-implied source span along the latent timeline. Second, the predicted clean latent already organizes coupled speech and visual articulation, allowing their temporal placement to be edited without regenerating the content. Building on these discoveries, we propose TimeSteer, a training-free framework that localizes each utterance's source span through Source Span Localization and transfers the associated audio-visual latent content from the source interval to the specified target interval through Region-Aware Latent Remapping. We further introduce SpeechShift, the first benchmark for interval-level speech scheduling in joint audio-visual generation. Experiments across two representative backbones show that TimeSteer substantially improves interval controllability over training-free baselines while maintaining competitive overall generation quality.