mlsanityData to 7 Oct 2026

Florence: A New Foundation Model for Computer Vision

arXiv 2111.11432 ↗First posted 22 Nov 2021cs.CV cs.AI cs.LGcited by 802

Not on the map. Why?

Cited by 208 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Automated visual understanding of our diverse and open world demands computer vision models to generalize well with minimal customization for specific tasks, similar to human vision. Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications. While existing vision foundation models such as CLIP, ALIGN, and Wu Dao 2.0 focus mainly on mapping images and textual representations to a cross-modal shared representation, we introduce a new computer vision foundation model, Florence, to expand the representations from coarse (scene) to fine (object), from static (images) to dynamic (videos), and from RGB to multiple modalities (caption, depth). By incorporating universal visual-language representations from Web-scale image-text data, our Florence model can be easily adapted for various computer vision tasks, such as classification, retrieval, object detection, VQA, image caption, video retrieval and action recognition. Moreover, Florence demonstrates outstanding performance in many types of transfer learning: fully sampled fine-tuning, linear probing, few-shot transfer and zero-shot transfer for novel images and objects. All of these properties are critical for our vision foundation model to serve general purpose vision tasks. Florence achieves new state-of-the-art results in majority of 44 representative benchmarks, e.g., ImageNet-1K zero-shot classification with top-1 accuracy of 83.74 and the top-5 accuracy of 97.18, 62.4 mAP on COCO fine tuning, 80.36 on VQA, and 87.8 on Kinetics-600.

Key facts. Cited by 802 papers in the map (data through 7 Oct 2026). Builds on 69 of its 88 references that are in the map. First posted on arXiv 22 Nov 2021.

Builds on

69 of 88 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2021ViViT: A Video Vision Transformercited by 1,610
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  14. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  15. 2017Mixed Precision Trainingcited by 1,135
  16. 2021Video Swin Transformercited by 1,090
  17. 2021Scaling Vision Transformerscited by 993
  18. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  19. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  20. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  21. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  22. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  23. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  24. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  25. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  26. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  27. 2018Do Better ImageNet Models Transfer Better?cited by 560
  28. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  29. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  30. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  31. 2020Rethinking Pre-training and Self-trainingcited by 352
  32. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  33. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  34. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 315
  35. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  36. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  37. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  38. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 154; 2023: 299; 2024: 185; 2025: 115; 2026: 44.

802 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  11. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 851
38 more
  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  3. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  8. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  9. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  10. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  11. 2023Segment Everything Everywhere All at Oncecited by 546
  12. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  13. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  14. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  15. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  16. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  17. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  18. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  19. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  20. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  21. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  22. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  23. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  24. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  25. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  26. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  27. 2022Visual Classification via Description from Large Language Modelscited by 331
  28. 2023ClimaX: A foundation model for weather and climatecited by 325
  29. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  30. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  31. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  32. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  33. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  34. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  35. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  36. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  37. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  38. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263

50 most cited of 802 All 802 (JSON)