mlsanityData to 7 Oct 2026

Classification Done Right for Vision-Language Pre-Training

arXiv 2411.03313 ↗First posted 5 Nov 2024cs.CVcited by 10

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP. https://github.com/x-cls/superclass

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 78 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

65 of 78 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Image Captioners Are Scalable Vision Learners Too13 Jun

Other references · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  13. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  14. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  15. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  16. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  17. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  18. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  19. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  20. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  21. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  22. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  23. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  24. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  25. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  26. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  27. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  28. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  29. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  30. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  31. 2021Image Segmentation Using Text and Image Promptscited by 509
  32. 2021OCR-free Document Understanding Transformercited by 400
  33. 2020Asymmetric Loss For Multi-Label Classificationcited by 379
  34. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  35. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  36. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  37. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  38. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 93
  2. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 28
  3. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Modelscited by 9
  4. 2025SuperCLIP: CLIP with Simple Classification Supervisioncited by 4
  5. 2026Let ViT Speak: Generative Language-Image Pre-trainingcited by 2
  6. 2026GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modelingcited by 2
  7. 2026Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methodscited by 1
  8. 2026TuringViT: Making SOTA Vision Transformers Accessible to Allcited by 1
  9. 2026Token Entropy Regularization for Multi-modal Antenna Affiliation Identificationcited by 0
  10. 2026FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
  2. 2023AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
  3. 2025Can Synthetic Images Serve as Effective and Efficient Class Prototypes?
  4. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models
  5. 2023SILC: Improving Vision Language Pretraining with Self-Distillation
  6. 2023UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding
  7. 2024BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models
  8. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries