mlsanityData to 14 Oct 2026

Towards Semantic Equivalence of Tokenization in Multimodal LLM

arXiv 2406.05127 ↗First posted 7 Jun 2024cs.CVcited by 45

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in processing vision-language tasks. One of the crux of MLLMs lies in vision tokenization, which involves efficiently transforming input visual signals into feature representations that are most beneficial for LLMs. However, existing vision tokenizers, essential for semantic alignment between vision and language, remain problematic. Existing methods aggressively fragment visual input, corrupting the visual semantic integrity. To address this, this paper proposes a novel dynamic Semantic-Equivalent Vision Tokenizer (SeTok), which groups visual features into semantic units via a dynamic clustering algorithm, flexibly determining the number of tokens based on image complexity. The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features. The proposed MLLM (Setokim) equipped with SeTok significantly demonstrates superior performance across various tasks, as evidenced by our experimental results. The project page is at https://chocowu.github.io/SeTok-web/.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 106 of its 127 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

106 of 127 references.

References · 106

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  10. 2023Segment Anythingcited by 8,234
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2016Layer Normalizationcited by 5,669
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  13. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  14. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  15. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  16. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  17. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  18. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  19. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  20. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  21. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  22. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  23. 2016Modeling Context in Referring Expressionscited by 1,063
  24. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  25. 2022Token Merging: Your ViT But Fastercited by 1,021
  26. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  27. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  28. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  29. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  30. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  31. 2020Object-Centric Learning with Slot Attentioncited by 876
  32. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  33. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  34. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  35. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  36. 2024Yi: Open Foundation Models by 01.AIcited by 719
  37. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  38. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 18; 2026: 10.

45 papers.

Sort:

By area:

  1. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  2. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 108
  3. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 61
  4. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  5. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  6. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  7. 2025AToken: A Unified Tokenizer for Visioncited by 30
  8. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  9. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuningcited by 22
  10. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimmingcited by 20
  11. 2024Towards Unified Multimodal Editing with Enhanced Knowledge Collaborationcited by 18
  12. 2025Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inferencecited by 18
33 more
  1. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  2. 2025JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generationcited by 15
  3. 2025Multi-Granular Multimodal Clue Fusion for Meme Understandingcited by 14
  4. 2024PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysiscited by 13
  5. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  6. 2024RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentationcited by 11
  7. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  8. 2024Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoningcited by 10
  9. 2025Position: Foundation Models Need Digital Twin Representationscited by 10
  10. 2024Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillationcited by 9
  11. 2025NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tablescited by 8
  12. 2025Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLMcited by 7
  13. 2026Towards Pixel-Level VLM Perception via Simple Points Predictioncited by 5
  14. 2024Measuring Agreeableness Bias in Multimodal Modelscited by 4
  15. 2024The Narrow Gate: Localized Image-Text Communication in Native Multimodal Modelscited by 4
  16. 2026A More Word-like Image Tokenization for MLLMscited by 3
  17. 2026V-RAE: Rethinking Video Latent Spaces for Generationcited by 3
  18. 2025QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retrainingcited by 2
  19. 2025Discrete JEPA: Learning Discrete Token Representations without Reconstructioncited by 2
  20. 2025LEAF-Mamba: Local Emphatic and Adaptive Fusion State Space Model for RGB-D Salient Object Detectioncited by 2
  21. 2024Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Imagecited by 1
  22. 2026Twins: Learn to Predict Unified Representations with Focal Losscited by 1
  23. 2025CalliReader: Contextualizing Chinese Calligraphy via an Embedding-Aligned Vision-Language Modelcited by 0
  24. 2025AlphaSpace: Enabling Robotic Actions through Semantic Tokenization and Symbolic Reasoningcited by 0
  25. 2025When Transformers Meet Recommenders: Integrating Self-Attentive Sequential Recommendation with Fine-Tuned LLMscited by 0
  26. 2025Harnessing Object Grounding for Time-Sensitive Video Understandingcited by 0
  27. 2025Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inferencecited by 0
  28. 2026Synergizing Understanding and Generation with Interleaved Analyzing-Drafting Thinkingcited by 0
  29. 2026SCP: Spatial Causal Prediction in Videocited by 0
  30. 2026Detached Skip-Links and R-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCRcited by 0
  31. 2026Expressivity In Multimodal Contrastive Learningcited by 0
  32. 2026When Do Frozen VLMs Respond to Image-Free Object-Token Edits? An Answer-Key-Free Protocol and What It Revealscited by 0
  33. 2026Pay More Attention To Text In High-Resolution MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Semantic Alignment for Multimodal Large Language Models
  2. 2025Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Models
  3. 2026Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
  4. 2024Efficient Multi-modal Large Language Models via Visual Token Grouping
  5. 2023What Makes for Good Visual Tokenizers for Large Language Models?
  6. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
  7. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See
  8. 2026TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications