mlsanityData to 7 Oct 2026

LAVIS: A Library for Language-Vision Intelligence

arXiv 2209.09019 ↗First posted 15 Sep 2022cs.CV cs.CL cs.LGcited by 55

Not on the map. Why?

Abstract (arXiv)

We introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications. LAVIS aims to serve as a one-stop comprehensive library that brings recent advancements in the language-vision field accessible for researchers and practitioners, as well as fertilizing future research and development. It features a unified interface to easily access state-of-the-art image-language, video-language models and common datasets. LAVIS supports training, evaluation and benchmarking on a rich variety of tasks, including multimodal classification, retrieval, captioning, visual question answering, dialogue and pre-training. In the meantime, the library is also highly extensible and configurable, facilitating future development and customization. In this technical report, we describe design principles, key components and functionalities of the library, and also present benchmarking results across common language-vision tasks. The library is available at: https://github.com/salesforce/LAVIS.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

45 of 59 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
33 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2017Localizing Moments in Video with Natural Languagecited by 660
  13. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  14. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  15. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  16. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  17. 2018nocaps: novel object captioning at scalecited by 444
  18. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  19. 2016Visual Dialogcited by 377
  20. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  21. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  22. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  23. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  24. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  25. 2019Audio-Visual Scene-Aware Dialogcited by 122
  26. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  27. 2019Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systemscited by 52
  28. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  29. 2021Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledgecited by 24
  30. 2020Video-Grounded Dialogues with Pretrained Generation Language Modelscited by 16
  31. 2021Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialoguescited by 11
  32. 2021VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Taskscited by 11
  33. 2021X-modaler: A Versatile and High-performance Codebase for Cross-modal Analyticscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 23; 2024: 19; 2025: 7; 2026: 3.

55 papers.

Sort:

By area:

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  4. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  5. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  6. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  7. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  8. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursingcited by 80
  9. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  10. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  11. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abusecited by 50
  12. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
38 more
  1. 2024EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuningcited by 42
  2. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  3. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformerscited by 21
  4. 2023PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Expertscited by 19
  5. 2023WebVLN: Vision-and-Language Navigation on Websitescited by 19
  6. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitaskscited by 19
  7. 2023Retrieval-based Knowledge Augmented Vision Language Pre-trainingcited by 17
  8. 2024VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimizationcited by 16
  9. 2023Cross-Domain Image Captioning with Discriminative Finetuningcited by 14
  10. 2024ControlCap: Controllable Region-level Captioningcited by 13
  11. 2024Multimodal Learned Sparse Retrieval with Probabilistic Expansion Controlcited by 12
  12. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  13. 2025FREE: Fast and Robust Vision Language Models with Early Exitscited by 12
  14. 2023Zero-shot Visual Question Answering with Language Model Feedbackcited by 11
  15. 2024The Role of Foundation Models in Neuro-Symbolic Learning and Reasoningcited by 11
  16. 2023Leveraging VLM-Based Pipelines to Annotate 3D Objectscited by 10
  17. 2024Question-Instructed Visual Descriptions for Zero-Shot Video Question Answeringcited by 8
  18. 2023Cloth2Tex: A Customized Cloth Texture Generation Pipeline for 3D Virtual Try-Oncited by 7
  19. 2024A LoRA is Worth a Thousand Picturescited by 7
  20. 2024Clinical Insights: A Comprehensive Review of Language Models in Medicinecited by 6
  21. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Modelscited by 6
  22. 2023Vision Language Transformers: A Surveycited by 5
  23. 2023Let's ViCE! Mimicking Human Cognitive Behavior in Image Generation Evaluationcited by 5
  24. 2024Identifying Crucial Objects in Blind and Low-Vision Individuals' Navigationcited by 5
  25. 2025ChartCards: A Chart-Metadata Generation Framework for Multi-Task Chart Understandingcited by 5
  26. 2023IMAD: IMage-Augmented multi-modal Dialoguecited by 4
  27. 2024A Dataset for Crucial Object Recognition in Blind and Low-Vision Individuals' Navigationcited by 4
  28. 2024DreamBarbie: Text to Barbie-Style 3D Avatarscited by 4
  29. 2024DynRefer: Delving into Region-level Multimodal Tasks via Dynamic Resolutioncited by 3
  30. 2025Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgettingcited by 3
  31. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 3
  32. 2023HADA: A Graph-based Amalgamation Framework in Image-text Retrievalcited by 2
  33. 2024R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generationcited by 2
  34. 2023When Parameter-efficient Tuning Meets General-purpose Vision-language Modelscited by 1
  35. 2024Shotluck Holmes: A Family of Efficient Small-Scale Large Language Vision Models For Video Captioning and Summarizationcited by 1
  36. 2025Argus: Leveraging Multiview Images for Improved 3-D Scene Understanding With Large Language Modelscited by 1
  37. 2023SPOTS: Stable Placement of Objects with Reasoning in Semi-Autonomous Teleoperation Systemscited by 0
  38. 2023BLIP-Adapter: Parameter-Efficient Transfer Learning for Mobile Screenshot Captioningcited by 0

50 most cited of 55 All 55 (JSON)