mlsanityData to 7 Oct 2026

Temporally Consistent Transformers for Video Generation

arXiv 2210.02396 ↗First posted 5 Oct 2022cs.CV cs.AI cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

To generate accurate videos, algorithms have to understand the spatial and temporal dependencies in the world. Current algorithms enable accurate predictions over short horizons but tend to suffer from temporal inconsistencies. When generated content goes out of view and is later revisited, the model invents different content instead. Despite this severe limitation, no established benchmarks on complex data exist for rigorously evaluating video generation with long temporal dependencies. In this paper, we curate 3 challenging video datasets with long-range dependencies by rendering walks through 3D scenes of procedural mazes, Minecraft worlds, and indoor scans. We perform a comprehensive evaluation of current models and observe their limitations in temporal consistency. Moreover, we introduce the Temporally Consistent Transformer (TECO), a generative model that substantially improves long-term consistency while also reducing sampling time. By compressing its input sequence into fewer embeddings, applying a temporal transformer, and expanding back using a spatial MaskGit, TECO outperforms existing models across many metrics. Videos are available on the website: https://wilson1yan.github.io/teco

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 57 of its 67 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

57 of 67 references.

References · 57

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2022Video Diffusion Modelscited by 2,280
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2020Rethinking Attention with Performerscited by 1,584
38 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  5. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  6. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  8. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  9. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  10. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  11. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  12. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  13. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  14. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  15. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  16. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  17. 2018Stochastic Video Generation with a Learned Priorcited by 244
  18. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  19. 2017Stochastic Variational Video Predictioncited by 234
  20. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  21. 2019Scaling Autoregressive Video Modelscited by 189
  22. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  23. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  24. 2018Stochastic Adversarial Video Predictioncited by 173
  25. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  26. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  27. 2022Generating Long Videos of Dynamic Scenescited by 113
  28. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  29. 2021An Attention Free Transformercited by 111
  30. 2016Video Pixel Networkscited by 108
  31. 2022Block-Recurrent Transformerscited by 107
  32. 2020Latent Video Transformercited by 106
  33. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  34. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  35. 2019High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networkscited by 88
  36. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  37. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  38. 2020Transformation-based Adversarial Video Prediction on Large-Scale Datacited by 66

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 16; 2025: 16; 2026: 9.

43 papers.

Sort:

By area:

  1. 2024Genie: Generative Interactive Environmentscited by 706
  2. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  4. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  5. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 178
  6. 2025History-Guided Video Diffusioncited by 165
  7. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  8. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  9. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  10. 2024Rolling Diffusion Modelscited by 102
  11. 2025Mixture of Contexts for Long Video Generationcited by 90
  12. 2024Video as the New Language for Real-World Decision Makingcited by 86
31 more
  1. 2025Long-Context State-Space Video World Modelscited by 59
  2. 2024On the Content Bias in Fr'echet Video Distancecited by 51
  3. 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraftcited by 46
  4. 2024A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddingscited by 23
  5. 2024Beyond Deepfake Images: Detecting AI-Generated Videoscited by 22
  6. 2025Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Frameworkcited by 20
  7. 2025MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generationcited by 18
  8. 2024UniCode: Learning a Unified Codebook for Multimodal Large Language Modelscited by 14
  9. 2025MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generationcited by 14
  10. 2023Understanding Physical Dynamics with Counterfactual World Modelingcited by 11
  11. 2025Enhance-A-Video: Better Generated Video for Freecited by 11
  12. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  13. 2024SEED4D: A Synthetic Ego--Exo Dynamic 4D Data Generator, Driving Dataset and Benchmarkcited by 8
  14. 2026Generative World Renderercited by 8
  15. 2024Precipitation Nowcasting Using Physics Informed Discriminator Generative Modelscited by 7
  16. 2025Accurate and Efficient World Modeling with Masked Latent Transformerscited by 7
  17. 2024Extreme Precipitation Nowcasting using Transformer-based Generative Modelscited by 6
  18. 2024Efficient Long Video Tokenization via Coordinate-based Patch Reconstructioncited by 6
  19. 2025A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Qualitycited by 6
  20. 2026MEG-XL: Data-Efficient Brain-to-Text via Long-Context Pre-Trainingcited by 6
  21. 2026Temporal Consistency-Aware Text-to-Motion Generationcited by 5
  22. 2025Toward Stable World Models: Measuring and Addressing World Instability in Generative Environmentscited by 4
  23. 2026Causality in Video Diffusers is Separable from Denoisingcited by 4
  24. 2026Flow Matching in Feature Space for Stochastic World Modelingcited by 4
  25. 2026Composition of Memory Experts for Diffusion World Modelscited by 3
  26. 2025Discrete JEPA: Learning Discrete Token Representations without Reconstructioncited by 2
  27. 20264DEquine: Disentangling Motion and Appearance for 4D Equine Reconstruction from Monocular Videocited by 2
  28. 2025GriDiT: Factorized Grid-Based Diffusion for Efficient Long Image Sequence Generationcited by 1
  29. 2026MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generationcited by 1
  30. 2025LiDAR-BIND-T: Improved and Temporally Consistent Sensor Modality Translation and Fusion for Robotic Applicationscited by 0
  31. 2026Equilibrium Forcing: Adaptive Video Generation Without Noise Conditioningcited by 0