mlsanityData to 14 Oct 2026

SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs

arXiv 2306.17842 ↗First posted 30 Jun 2023cs.CV cs.CL cs.MMcited by 55

Not on the map. Why?

Abstract (arXiv)

In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts between raw pixels and interpretable lexical tokens (or words) extracted from the LLM's vocabulary. The resulting tokens capture both the semantic meaning and the fine-grained details needed for visual reconstruction, effectively translating the visual content into a language comprehensible to the LLM, and empowering it to perform a wide array of multimodal tasks. Our approach is validated through in-context learning experiments with frozen PaLM 2 and GPT 3.5 on a diverse set of image understanding and generation tasks. Our method marks the first successful attempt to enable a frozen LLM to generate image content while surpassing state-of-the-art performance in image understanding tasks, under the same setting, by over 25%.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 45 of its 52 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

45 of 52 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2023GPT-4 Technical Reportcited by 18,396
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  10. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  11. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
33 more
  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2016Improved Techniques for Training GANscited by 2,970
  9. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  10. 2022Emergent Abilities of Large Language Modelscited by 2,512
  11. 2016Matching Networks for One Shot Learningcited by 2,255
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  13. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  14. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  15. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  16. 2023PaLM 2 Technical Reportcited by 1,162
  17. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  18. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  19. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  20. 2022High Fidelity Neural Audio Compressioncited by 687
  21. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  22. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  23. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  24. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  25. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  26. 2022MAGVIT: Masked Generative Video Transformercited by 346
  27. 2018A Short Note about Kinetics-600cited by 333
  28. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  29. 2015Exploring Models and Data for Image Question Answeringcited by 139
  30. 2021Regularizing Generative Adversarial Networks under Limited Datacited by 107
  31. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  32. 2021Cross-Modal Discrete Representation Learningcited by 35
  33. 2023Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignmentcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 30; 2025: 16; 2026: 3.

55 papers.

Sort:

By area:

  1. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  2. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 366
  3. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 102
  4. 2024Loong: Generating Minute-level Long Videos with Autoregressive Language Modelscited by 99
  5. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 94
  6. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  7. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  8. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  9. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 57
  10. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 56
  11. 2024Autoregressive Models in Vision: A Surveycited by 39
  12. 2024Language-Guided Image Tokenization for Generationcited by 36
38 more
  1. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  2. 2024Transfer CLIP for Generalizable Image Denoisingcited by 29
  3. 2025One-Step Diffusion-Based Image Compression with Semantic Distillationcited by 26
  4. 2024Beyond Text: Frozen Large Language Models in Visual Signal Comprehensioncited by 23
  5. 2025DLF: Extreme Image Compression with Dual-generative Latent Fusioncited by 21
  6. 2024Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modelingcited by 16
  7. 2024Mimir: Improving Video Diffusion Models for Precise Text Understandingcited by 16
  8. 2024VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modelingcited by 15
  9. 2023De-Diffusion Makes Text a Strong Cross-Modal Interfacecited by 14
  10. 2024UniCode: Learning a Unified Codebook for Multimodal Large Language Modelscited by 14
  11. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  12. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  13. 2024XQ-GAN: An Open-source Image Tokenization Framework for Autoregressive Generationcited by 12
  14. 2024Visual Lexicon: Rich Image Features in Language Spacecited by 12
  15. 2024SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretizationcited by 12
  16. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  17. 2025Image Tokenizer Needs Post-Trainingcited by 12
  18. 2024What Makes for Good Image Captions?cited by 11
  19. 2024Factorized Visual Tokenization and Generationcited by 11
  20. 2024LG-VQ: Language-Guided Codebook Learningcited by 10
  21. 2025Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesiscited by 10
  22. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  23. 2023TEAL: Tokenize and Embed ALL for Multi-modal Large Language Modelscited by 7
  24. 2024Low-dose CT Denoising with Language-engaged Dual-space Alignmentcited by 7
  25. 2024Multi-View Empowered Structural Graph Wordification for Language Modelscited by 7
  26. 2024Multi-Scale Fusion for Object Representationcited by 7
  27. 2023Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generationcited by 6
  28. 2025V2Flow: Unifying Visual Tokenization and Large Language Model Vocabularies for Autoregressive Image Generationcited by 6
  29. 2026ResTok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generationcited by 5
  30. 2024Scaling Image Tokenizers with Grouped Spherical Quantizationcited by 4
  31. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Modelscited by 4
  32. 2025Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Textcited by 4
  33. 2025Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Modelscited by 4
  34. 2024Can Large Language Models Learn Independent Causal Mechanisms?cited by 3
  35. 2025A Survey: Spatiotemporal Consistency in Video Generationcited by 3
  36. 2025LangMamba: A Language-driven Mamba Framework for Low-dose CT Denoising with Vision-language Modelscited by 3
  37. 2025GR-LLMs: Recent Advances in Generative Recommendation Based on Large Language Modelscited by 3
  38. 2024GiVE: Guiding Visual Encoder to Perceive Overlooked Informationcited by 2

50 most cited of 55 All 55 (JSON)