mlsanityData to 14 Oct 2026

SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretization

arXiv 2412.10443 ↗First posted 11 Dec 2024cs.CV cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper presents the Semantic-aWarE spatial-tEmporal Tokenizer (SweetTok), a novel video tokenizer to overcome the limitations in current video tokenization methods for compacted yet effective discretization. Unlike previous approaches that process flattened local visual patches via direct discretization or adaptive query tokenization, SweetTok proposes a decoupling framework, compressing visual inputs through distinct spatial and temporal queries via Decoupled Query AutoEncoder (DQAE). This design allows SweetTok to efficiently compress video token count while achieving superior fidelity by capturing essential information across spatial and temporal dimensions. Furthermore, we design a Motion-enhanced Language Codebook (MLC) tailored for spatial and temporal compression to address the differences in semantic representation between appearance and motion information. SweetTok significantly improves video reconstruction results by 42.8% w.r.t rFVD on UCF-101 dataset. With a better token compression strategy, it also boosts downstream video generation results by 15.1% w.r.t gFVD. Additionally, the compressed decoupled tokens are imbued with semantic information, enabling few-shot recognition capabilities powered by LLMs in downstream applications.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 45 of its 56 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

45 of 56 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  10. 2020End-to-End Object Detection with Transformerscited by 7,414
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
33 more
  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  4. 2017The Kinetics Human Action Video Datasetcited by 1,909
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  6. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  9. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  10. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  11. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  12. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  13. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  14. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  15. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  16. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  17. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  18. 2022MAGVIT: Masked Generative Video Transformercited by 346
  19. 2018A Short Note about Kinetics-600cited by 333
  20. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  21. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  22. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  23. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 118
  24. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100
  25. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  26. 2023SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMscited by 55
  27. 2024LARP: Tokenizing Videos with a Learned Autoregressive Generative Priorcited by 46
  28. 2023Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignmentcited by 29
  29. 2024Beyond Text: Frozen Large Language Models in Visual Signal Comprehensioncited by 23
  30. 2024OmniVid: A Generative Framework for Universal Video Understandingcited by 22
  31. 2024Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modelingcited by 16
  32. 2023De-Diffusion Makes Text a Strong Cross-Modal Interfacecited by 14
  33. 2021Few Shot Activity Recognition Using Variational Inferencecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  2. 2025DINO-Tok: Adapting DINO for Visual Tokenizerscited by 6
  3. 2025DeRA: Decoupled Representation Alignment for Video Tokenizationcited by 6
  4. 2026PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generationcited by 4
  5. 2026TaskTok: Delving into Task Tokens for Task-driven Image Restorationcited by 3
  6. 2026VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoderscited by 3
  7. 2026V-RAE: Rethinking Video Latent Spaces for Generationcited by 3
  8. 2025Versatile Video Representation via Feed-Forward 2D Gaussian Splatting Tokenizationcited by 1
  9. 2026Best of Both Worlds: Multimodal Reasoning and Generation via Unified Discrete Flow Matchingcited by 1
  10. 2026PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editingcited by 1
  11. 2026TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenizationcited by 1
  12. 2026Keep-or-Drop? Adaptive Tokenizer for Compact Video Representationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VideoTok4D: A 4D-Aware Video Tokenizer for Compact World Representation
  2. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models
  3. 2025One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory
  4. 2025TVC: Tokenized Video Compression with Ultra-Low Bit Rate
  5. 2025DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Foundation Models
  6. 2026Beyond Selection: Token Parameterization for Extreme Visual Token Compression
  7. 2026Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models
  8. 2025RefTok: Reference-Based Tokenization for Video Generation