mlsanityData to 7 Oct 2026

OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation

arXiv 2406.09399 ↗First posted 13 Jun 2024cs.CVcited by 117

Not on the map. Why?

Cited by 24 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Tokenizer, serving as a translator to map the intricate visual data into a compact latent space, lies at the core of visual generative models. Based on the finding that existing tokenizers are tailored to image or video inputs, this paper presents OmniTokenizer, a transformer-based tokenizer for joint image and video tokenization. OmniTokenizer is designed with a spatial-temporal decoupled architecture, which integrates window and causal attention for spatial and temporal modeling. To exploit the complementary nature of image and video data, we further propose a progressive training strategy, where OmniTokenizer is first trained on image data on a fixed resolution to develop the spatial encoding capacity and then jointly trained on image and video data on multiple resolutions to learn the temporal dynamics. OmniTokenizer, for the first time, handles both image and video inputs within a unified framework and proves the possibility of realizing their synergy. Extensive experiments demonstrate that OmniTokenizer achieves state-of-the-art (SOTA) reconstruction performance on various image and video datasets, e.g., 1.11 reconstruction FID on ImageNet and 42 reconstruction FVD on UCF-101, beating the previous SOTA methods by 13% and 26%, respectively. Additionally, we also show that when integrated with OmniTokenizer, both language model-based approaches and diffusion models can realize advanced visual synthesis performance, underscoring the superiority and versatility of our method. Code is available at https://github.com/FoundationVision/OmniTokenizer.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 63 of its 71 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

63 of 71 references.

References · 63

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  12. 2022Scalable Diffusion Models with Transformerscited by 5,931
38 more
  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  12. 2017The Kinetics Human Action Video Datasetcited by 1,894
  13. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  14. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  15. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  16. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  17. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  18. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  19. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  20. 2021Scaling Vision Transformerscited by 993
  21. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  22. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  23. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  24. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  25. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  26. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  27. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  28. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  29. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  30. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  31. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  32. 2018A Short Note about Kinetics-600cited by 331
  33. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  34. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  35. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  36. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  37. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  38. 2019Scaling Autoregressive Video Modelscited by 189

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 56; 2026: 41.

117 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  3. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 153
  4. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  5. 2025Controllable Video Generation: A Surveycited by 93
  6. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  7. 2025Improving the Diffusability of Autoencoderscited by 74
  8. 2024Parallelized Autoregressive Visual Generationcited by 58
  9. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  10. 2024LARP: Tokenizing Videos with a Learned Autoregressive Generative Priorcited by 46
  11. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformercited by 45
  12. 2025InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generationcited by 43
38 more
  1. 2024Autoregressive Models in Vision: A Surveycited by 39
  2. 2024VidTok: A Versatile and Open-Source Video Tokenizercited by 35
  3. 2025VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual interventioncited by 33
  4. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  5. 2025AToken: A Unified Tokenizer for Visioncited by 30
  6. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  7. 2025Taming Teacher Forcing for Masked Autoregressive Video Generationcited by 29
  8. 2025Neighboring Autoregressive Modeling for Efficient Visual Generationcited by 29
  9. 2025Next Block Prediction: Video Generation via Semi-Autoregressive Modelingcited by 26
  10. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 25
  11. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  12. 2025Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspectivecited by 22
  13. 2024LaVin-DiT: Large Vision Diffusion Transformercited by 21
  14. 2025PhysiX: A Foundation Model for Physics Simulationscited by 21
  15. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenizationcited by 21
  16. 2025The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencodingcited by 21
  17. 2024Epsilon-VAE: Denoising as Visual Decodingcited by 16
  18. 2025Generative Pre-trained Autoregressive Diffusion Transformercited by 16
  19. 2025Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Spacecited by 16
  20. 2025LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Modelscited by 15
  21. 2025DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencodercited by 15
  22. 2024REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latentscited by 14
  23. 2024When Worse is Better: Navigating the compression-generation tradeoff in visual tokenizationcited by 14
  24. 2024SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretizationcited by 12
  25. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  26. 2025H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Modelscited by 12
  27. 2025TokBench: Evaluating Your Visual Tokenizer before Visual Generationcited by 12
  28. 2024Factorized Visual Tokenization and Generationcited by 11
  29. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  30. 2025Anchor Token Matching: Implicit Structure Locking for Training-free AR Image Editingcited by 10
  31. 2025Position: Foundation Models Need Digital Twin Representationscited by 10
  32. 2025Rethinking Video Tokenization: A Conditioned Diffusion-based Approachcited by 9
  33. 2025MotionStrata: Hierarchical Motion Latents for Compact Video Autoencodingcited by 9
  34. 2025WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstructioncited by 9
  35. 2026PISCO: Precise Video Instance Insertion with Sparse Controlcited by 9
  36. 2026RepWAM: World Action Modeling with Representation Visual-Action Tokenizerscited by 9
  37. 2025CODA: Repurposing Continuous VAEs for Discrete Tokenizationcited by 8
  38. 2025Video-GPT via Next Clip Diffusioncited by 8

50 most cited of 117 All 117 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Learnings from Scaling Visual Tokenizers for Reconstruction and Generation
  2. 2025UniTok: A Unified Tokenizer for Visual Generation and Understanding
  3. 2024An Image is Worth 32 Tokens for Reconstruction and Generation
  4. 2025Omni-Video: Democratizing Unified Video Understanding and Generation
  5. 2024Towards Multi-Task Multi-Modal Models: A Video Generative Perspective
  6. 2026Communication-Inspired Tokenization for Structured Image Representations
  7. 2025Adapting Self-Supervised Representations as a Latent Space for Efficient Generation
  8. 2022MAGVIT: Masked Generative Video Transformer