mlsanityData to 7 Oct 2026

From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities

arXiv 2410.02155 ↗First posted 3 Oct 2024cs.AI cs.CL cs.CVcited by 16

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities. We introduce a novel image tokenizer that bridges this gap by applying the principle of Byte-Pair Encoding (BPE) to visual data. Unlike conventional approaches that rely on separate visual encoders, our method directly incorporates structural prior information into image tokens, mirroring the successful tokenization strategies used in text-only Large Language Models. This innovative approach enables Transformer models to more effectively learn and reason across modalities. Through theoretical analysis and extensive experiments, we demonstrate that our BPE Image Tokenizer significantly enhances MLLMs' multimodal understanding capabilities, even with limited training data. Leveraging this method, we develop Being-VL-0, a model that demonstrates superior performance across various benchmarks and shows promising scalability, potentially paving the way for more efficient and capable multimodal foundation models.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 47 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

38 of 47 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
26 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  8. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  11. 2023A Survey on Multimodal Large Language Modelscited by 898
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  13. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  14. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  15. 2023Instruction Tuning with GPT-4cited by 676
  16. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  17. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  18. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  19. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  20. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  21. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  22. 2024Language Models Still Struggle to Zero-shot Reason about Time Seriescited by 81
  23. 2023Vision Transformer with Quadrangle Attentioncited by 46
  24. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 39
  25. 2024Toward a Theory of Tokenization in LLMscited by 28
  26. 2024UniCode: Learning a Unified Codebook for Multimodal Large Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  3. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  4. 2024Scaling Large Motion Models with Million-Level Human Motionscited by 20
  5. 2024VideoOrion: Tokenizing Object Dynamics in Videoscited by 12
  6. 2026ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Modelcited by 12
  7. 2025Unified Multimodal Understanding via Byte-Pair Visual Encodingcited by 9
  8. 2025Probing Perceptual Constancy in Large Vision-Language Modelscited by 7
  9. 2026Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularizationcited by 7
  10. 2025GestureHYDRA: Semantic Co-speech Gesture Synthesis via Hybrid Modality Diffusion Transformer and Cascaded-Synchronized Retrieval-Augmented Generationcited by 6
  11. 2025Transport Discrepancy as a Reliability Signal for Vision-Language-Action Modelscited by 6
  12. 2024Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generationcited by 2
4 more
  1. 2026BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encodercited by 1
  2. 2026OpenT2M: No-frill Motion Generation with Open-source,Large-scale, High-quality Datacited by 1
  3. 2026Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Modelscited by 1
  4. 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Planting a SEED of Vision in Large Language Model
  2. 2026A More Word-like Image Tokenization for MLLMs
  3. 2026What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
  4. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
  5. 2025ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task
  6. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
  7. 2025Token Sequence Compression for Efficient Multimodal Computing
  8. 2025UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation