mlsanityData to 7 Oct 2026

ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generation

arXiv 2505.16495 ↗First posted 22 May 2025cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

While humans effortlessly draw visual objects and shapes by adaptively allocating attention based on their complexity, existing multimodal large language models (MLLMs) remain constrained by rigid token representations. Bridging this gap, we propose ALTo, an adaptive length tokenizer for autoregressive mask generation. To achieve this, a novel token length predictor is designed, along with a length regularization term and a differentiable token chunking strategy. We further build ALToLLM that seamlessly integrates ALTo into MLLM. Preferences on the trade-offs between mask quality and efficiency is implemented by group relative policy optimization (GRPO). Experiments demonstrate that ALToLLM achieves state-of-the-art performance with adaptive token cost on popular segmentation benchmarks. Code and models are released at https://github.com/yayafengzi/ALToLLM.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 66 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

59 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model17 Mar

Other references · 58

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Segment Anythingcited by 8,158
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
38 more
  1. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  2. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  4. 2016Modeling Context in Referring Expressionscited by 1,054
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  7. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  8. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  9. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  10. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  11. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  12. 2023Segment Everything Everywhere All at Oncecited by 546
  13. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  14. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  15. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  16. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  17. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  18. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  19. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  20. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  21. 2022Visual Prompting via Image Inpaintingcited by 248
  22. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
  23. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  24. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  25. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  26. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  27. 2025FlexTok: Resampling Images into 1D Token Sequences of Flexible Lengthcited by 105
  28. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  29. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  30. 2022UViM: A Unified Modeling Approach for Vision with Learned Guiding Codescited by 77
  31. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  32. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  33. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  34. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  35. 2023LLaFS: When Large Language Models Meet Few-Shot Segmentationcited by 48
  36. 2024ElasticTok: Adaptive Tokenization for Image and Videocited by 44
  37. 2024Adaptive Length Image Tokenization via Recurrent Allocationcited by 40
  38. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
  2. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 8
  3. 2026EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generationcited by 7
  4. 2026Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPOcited by 1
  5. 2026OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs
  2. 2025Token Sequence Compression for Efficient Multimodal Computing
  3. 2026One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs
  4. 2026When LLaVA Meets Objects: Token Composition for Vision-Language-Models
  5. 2026Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation
  6. 2025TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contexts
  7. 2026Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models
  8. 2025Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models