mlsanityData to 14 Oct 2026

Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language

arXiv 2306.16410 ↗First posted 28 Jun 2023cs.CL cs.CVcited by 53

Not on the map. Why?

Abstract (arXiv)

We propose LENS, a modular approach for tackling computer vision problems by leveraging the power of large language models (LLMs). Our system uses a language model to reason over outputs from a set of independent and highly descriptive vision modules that provide exhaustive information about an image. We evaluate the approach on pure computer vision settings such as zero- and few-shot object recognition, as well as on vision and language problems. LENS can be applied to any off-the-shelf LLM and we find that the LLMs with LENS perform highly competitively with much bigger and much more sophisticated systems, without any multimodal training whatsoever. We open-source our code at https://github.com/ContextualAI/lens and provide an interactive demo.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 50 of its 64 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

50 of 64 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  12. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  13. 2018Hierarchical Neural Story Generationcited by 1,193
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  15. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  16. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  17. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  18. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  19. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  20. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  21. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  22. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  23. 2018nocaps: novel object captioning at scalecited by 444
  24. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  25. 2022Visual Classification via Description from Large Language Modelscited by 336
  26. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  27. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  28. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  29. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  30. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  31. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  32. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  33. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  34. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  35. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  36. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  37. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  38. 2022Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioningcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 30; 2025: 8; 2026: 2.

53 papers.

Sort:

By area:

  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  2. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  4. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  5. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  6. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 103
  7. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 62
  8. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 60
  9. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  10. 2023GATGPT: A Pre-trained Large Language Model with Graph Attention Network for Spatiotemporal Imputationcited by 35
  11. 2024Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Drivingcited by 35
  12. 2024How Can Large Language Models Understand Spatial-Temporal Data?cited by 33
38 more
  1. 2024LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Contentcited by 33
  2. 2024Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discoverycited by 31
  3. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  4. 2024Democratizing Fine-grained Visual Recognition with Large Language Modelscited by 25
  5. 2024No Culture Left Behind: ArtELingo-28, a Benchmark of WikiArt with Captions in 28 Languagescited by 19
  6. 2023Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Modelscited by 14
  7. 2023De-Diffusion Makes Text a Strong Cross-Modal Interfacecited by 14
  8. 2024GPT-4o: Visual perception performance of multimodal large language models in piglet activity understandingcited by 12
  9. 2024Leveraging Large Language Models for Multimodal Searchcited by 11
  10. 2024Similarity-based Neighbor Selection for Graph LLMscited by 10
  11. 2024Question-Instructed Visual Descriptions for Zero-Shot Video Question Answeringcited by 9
  12. 2024The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioningcited by 9
  13. 2023MIND: Multi-Task Incremental Network Distillationcited by 8
  14. 2024Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memescited by 8
  15. 2025How Can Objects Help Video-Language Understanding?cited by 8
  16. 2024TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language Navigationcited by 7
  17. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  18. 2025Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agentscited by 6
  19. 2024Improving Fine-grained Visual Understanding in VLMs through Text-Only Trainingcited by 5
  20. 2023Defining a New NLP Playgroundcited by 4
  21. 2024Image Safeguarding: Reasoning with Conditional Vision Language Model and Obfuscating Unsafe Content Counterfactuallycited by 4
  22. 2025RHYTHM: Reasoning with Hierarchical Temporal Tokenization for Human Mobilitycited by 4
  23. 2023Language as the Medium: Multimodal Video Classification through text onlycited by 3
  24. 2024VLLaVO: Mitigating Visual Gap through LLMscited by 2
  25. 2024Multi-Granularity Language-Guided Training for Multi-Object Trackingcited by 2
  26. 2024Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Modelscited by 2
  27. 2025Multi-aspect Knowledge Distillation with Large Language Modelcited by 2
  28. 2023Application of frozen large-scale models to multimodal task-oriented dialoguecited by 1
  29. 2023Leveraging Diffusion Perturbations for Measuring Fairness in Computer Visioncited by 1
  30. 2024Describing Images textitFast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processescited by 1
  31. 2024Intelligent Director: An Automatic Framework for Dynamic Visual Composition using ChatGPTcited by 1
  32. 2024Decoding Emotions in Abstract Art: Cognitive Plausibility of CLIP in Recognizing Color-Emotion Associationscited by 1
  33. 2024RITA: A Real-time Interactive Talking Avatars Frameworkcited by 1
  34. 2025Flexible Concept Bottleneck Modelcited by 1
  35. 2026Document Optimization for Black-Box Retrieval via Reinforcement Learningcited by 1
  36. 2026EVL-MCoT: Enhanced Vision-Language Multi-CoT for Harmful Meme Detectioncited by 1
  37. 2024Simplifying Multimodality: Unimodal Approach to Multimodal Challenges in Radiology with General-Domain Large Language Modelcited by 0
  38. 2024Target Prompting for Information Extraction with Vision Language Modelcited by 0

50 most cited of 53 All 53 (JSON)