mlsanityData to 14 Oct 2026

Sigmoid Loss for Language Image Pre-Training

arXiv 2303.15343 ↗First posted 27 Mar 2023cs.CV cs.AIcited by 3,109

Not on the map. Why?

Cited by 1,094 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP). Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days. The disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We release our models at https://github.com/google-research/big_vision and hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.

Key facts. Cited by 3,109 papers in the map (data through 14 Oct 2026). Builds on 42 of its 62 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

42 of 62 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2020Supervised Contrastive Learningcited by 2,829
30 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  6. 2018Model Cards for Model Reportingcited by 1,080
  7. 2021Scaling Vision Transformerscited by 999
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  9. 2021Robust fine-tuning of zero-shot modelscited by 797
  10. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  13. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  14. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  15. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  16. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  17. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  18. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  19. 2020Are we done with ImageNet?cited by 333
  20. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  21. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  22. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  23. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  24. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  25. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  26. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  27. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  28. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  29. 2022Better plain ViT baselines for ImageNet-1kcited by 98
  30. 2022CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNetcited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 52; 2024: 498; 2025: 1282; 2026: 1277.

3,109 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2025Gemma 3 Technical Reportcited by 1,824
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  6. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  9. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  10. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
  11. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  12. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
38 more
  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 530
  2. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  3. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  4. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  5. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  6. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  7. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  8. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  9. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  10. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  11. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 404
  12. 2025Kimi-VL Technical Reportcited by 388
  13. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 367
  14. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
  15. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  16. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  17. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  18. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  19. 2024What matters when building vision-language models?cited by 326
  20. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  21. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  22. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  23. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 304
  24. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  25. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  26. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  27. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  28. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  29. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  30. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  31. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  32. 2025Transfer between Modalities with MetaQueriescited by 236
  33. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  34. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  35. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  36. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  37. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  38. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209

50 most cited of 3,109 All 3,109 (JSON)