# Generative AI for Video Translation: A Scalable Architecture for Multilingual Video Conferencing

arXiv: https://arxiv.org/abs/2512.13904 · HTML: https://mlsanity.com/paper/2512.13904/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 55 references that are in the map.
- First posted on arXiv 15 Dec 2025.

## Builds on (most cited first, 21 of 21)

- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time](https://mlsanity.com/paper/2404.10667/) (2024)
- [XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model](https://mlsanity.com/paper/2406.04904/) (2024)
- [Depth-Aware Generative Adversarial Network for Talking Head Video Generation](https://mlsanity.com/paper/2203.06605/) (2022)
- [StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN](https://mlsanity.com/paper/2203.04036/) (2022)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation](https://mlsanity.com/paper/2301.03396/) (2023)
- [DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation](https://mlsanity.com/paper/2301.03786/) (2023)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models](https://mlsanity.com/paper/2502.01061/) (2025)
- [Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis](https://mlsanity.com/paper/2401.08503/) (2024)
- [Towards Automatic Face-to-Face Translation](https://mlsanity.com/paper/2003.00418/) (2020)
- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [VividTalk: One-Shot Audio-Driven Talking Head Generation Based on 3D Hybrid Prior](https://mlsanity.com/paper/2312.01841/) (2023)
- [GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation](https://mlsanity.com/paper/2305.00787/) (2023)
- [Lip to Speech Synthesis with Visual Context Attentional GAN](https://mlsanity.com/paper/2204.01726/) (2022)
- [Talking Face Generation with Multilingual TTS](https://mlsanity.com/paper/2205.06421/) (2022)
- [MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes](https://mlsanity.com/paper/2410.06734/) (2024)
- [Unet-TTS: Improving Unseen Speaker and Style Transfer in One-shot Voice Cloning](https://mlsanity.com/paper/2109.11115/) (2021)
- [TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation](https://mlsanity.com/paper/2312.15197/) (2023)
- [Synthetic Patients: Simulating Difficult Conversations with Multimodal Generative AI for Medical Education](https://mlsanity.com/paper/2405.19941/) (2024)

## Abstract (arXiv)

The real-time deployment of cascaded generative AI pipelines for applications like video translation is constrained by significant system-level challenges. These include the cumulative latency of sequential model inference and the quadratic (O(N2)) computational complexity that renders multi-user video conferencing applications unscalable. This paper proposes and evaluates a practical system-level framework designed to mitigate these critical bottlenecks. The proposed architecture incorporates a turn-taking mechanism to reduce computational complexity from quadratic to linear in multi-user scenarios, and a segmented processing protocol to manage inference latency for a perceptually real-time experience. We implement a proof-of-concept pipeline and conduct a rigorous performance analysis across a multi-tiered hardware setup, including commodity (NVIDIA RTX 4060), cloud (NVIDIA T4), and enterprise (NVIDIA A100) GPUs. Our objective evaluation demonstrates that the system achieves real-time throughput (τ < 1.0) on modern hardware. A subjective user study further validates the approach, showing that a predictable, initial processing delay is highly acceptable to users in exchange for a smooth, uninterrupted playback experience. The work presents a validated, end-to-end system design that offers a practical roadmap for deploying scalable, real-time generative AI applications in multilingual communication platforms.