mlsanityData to 14 Oct 2026

Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning

arXiv 2511.19515 ↗First posted 24 Nov 2025cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

This paper investigates the fundamental relationship between model capacity and the minimal number of visual tokens required to preserve image semantics. Inspired by the Minimum Description Length principle, we reinterpret image tokens as vectors in a visual semantic space and define the intrinsic semantic complexity of an image as the smallest set of basis vectors needed to span this space. Building on this perspective, we propose Orthogonal Filtering, a lightweight module that adaptively clusters redundant tokens into a compact set of orthogonal bases. Through extensive experiments across a range of ViT models, we reveal a consistent token, model scaling law: larger models require significantly fewer tokens to span visual semantic space. Besides, we also contribute a visual long-context dataset.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 43 of its 56 references that are in the map. First posted on arXiv 24 Nov 2025.

Builds on

43 of 56 references.

References · 43

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2017Mask R-CNNcited by 7,277
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  7. 2019Natural Adversarial Examplescited by 1,408
  8. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  11. 2021Scaling Vision Transformerscited by 999
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
31 more
  1. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  2. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  3. 2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
  4. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  5. 2020Proxy Anchor Loss for Deep Metric Learningcited by 195
  6. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  7. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  8. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 177
  9. 2022How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?cited by 128
  10. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 125
  11. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  12. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 81
  13. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  14. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77
  15. 2024When Do We Not Need Larger Vision Models?cited by 67
  16. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 60
  17. 2022MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformerscited by 47
  18. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  19. 2022On Data Scaling in Masked Image Modelingcited by 46
  20. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  21. 2025Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And Morecited by 28
  22. 2023Segmentation and Vascular Vectorization for Coronary Artery by Geometry-based Cascaded Neural Networkcited by 21
  23. 2024Towards Precise Scaling Laws for Video Diffusion Transformerscited by 20
  24. 2024Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwardscited by 19
  25. 2024LookupViT: Compressing visual information to a limited number of tokenscited by 17
  26. 2024One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representationcited by 16
  27. 2024Scaling Properties of Diffusion Models for Perceptual Taskscited by 15
  28. 2025Drift-aware Collaborative Assistance Mixture of Experts for Heterogeneous Multistream Learningcited by 14
  29. 2023Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Modelscited by 8
  30. 2025Turning Drift into Constraint: Robust Reasoning Alignment in Non-Stationary Multi-Stream Environmentscited by 5
  31. 2025Walking the Tightrope: Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026TC-SSA: Token Compression via Semantic Slot Aggregation for Gigapixel Pathology Reasoningcited by 17
  2. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  3. 2026Efficient Chest X-ray Representation Learning via Semantic-Partitioned Contrastive Learningcited by 10
  4. 2026Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Modelscited by 7
  5. 2026Multimodal Model for Computational Pathology:Representation Learning and Image Compressioncited by 6
  6. 2026A unified multi-task framework enables interpretable chest radiograph analysiscited by 6
  7. 2026XrayClaw: Cooperative-Competitive Multi-Agent Alignment for Trustworthy Chest X-ray Diagnosiscited by 5
  8. 2026Stepwise Token Selection for Efficient Multimodal Large Language Modelscited by 5
  9. 2026Learnable Token Sparsification for Efficient Gigapixel Whole Slide Image Reasoningcited by 4
  10. 2026Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuningcited by 2
  11. 2026PathSelect: Sequential Token Selection for Whole Slide Pathologycited by 2
  12. 2026Decoupling semantics from vision: A framework for faithful visual-text compression evaluationcited by 2
4 more
  1. 2026DiffPrune: differentiable information throttling for token pruning in vision-language modelscited by 1
  2. 2026From Snapshots to Symphonies: The Evolution of Protein Prediction from Static Structures to Generative Dynamics and Multimodal Interactionscited by 0
  3. 2026Autonomous Drift Learning in Data Streams: A Unified Perspectivecited by 0
  4. 2026StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Modelscited by 0