mlsanityData to 7 Oct 2026

Visual Large Language Models for Generalized and Specialized Applications

arXiv 2501.02765 ↗First posted 6 Jan 2025cs.CV cs.AIcited by 35

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Visual-language models (VLM) have emerged as a powerful tool for learning a unified embedding space for vision and language. Inspired by large language models, which have demonstrated strong reasoning and multi-task capabilities, visual large language models (VLLMs) are gaining increasing attention for building general-purpose VLMs. Despite the significant progress made in VLLMs, the related literature remains limited, particularly from a comprehensive application perspective, encompassing generalized and specialized applications across vision (image, video, depth), action, and language modalities. In this survey, we focus on the diverse applications of VLLMs, examining their using scenarios, identifying ethics consideration and challenges, and discussing future directions for their development. By synthesizing these contents, we aim to provide a comprehensive guide that will pave the way for future innovations and broader applications of VLLMs. The paper list repository is available: https://github.com/JackYFL/awesome-VLLMs.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 401 of its 458 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

401 of 458 references.

Closest ancestors · 32

Same direction, newest first

  1. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov
  2. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
  3. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?1 Jul
  4. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models25 Jun
  5. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models19 Apr
  6. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  7. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar
  8. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model21 Mar
  9. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentation26 Feb
  10. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions20 Feb
  11. 2024ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling9 Feb
  12. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model19 Dec
20 more
  1. 2023GSVA: Generalized Segmentation via Multimodal Large Language Models15 Dec
  2. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models11 Dec
  3. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models5 Dec
  4. 2023Lenna: Language Enhanced Reasoning Detection Assistant5 Dec
  5. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  6. 2023Griffon: Spelling out All Object Locations at Any Granularity with Large Language Models24 Nov
  7. 2023Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs22 Nov
  8. 2023NExT-Chat: An LMM for Chat, Detection and Segmentation8 Nov
  9. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  10. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  13. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions19 Aug
  14. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  15. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  16. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  17. 2023Contextual Object Detection with Multimodal Large Language Models29 May
  18. 2023Chain of Thought Prompt Tuning in Vision Language Models16 Apr
  19. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr
  20. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 369

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2022Training language models to follow instructions with human feedbackcited by 15,712
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  11. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  12. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
38 more
  1. 2023Segment Anythingcited by 8,158
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  13. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  14. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  15. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  16. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  17. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  18. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  19. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  20. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  21. 2015VQA: Visual Question Answeringcited by 2,489
  22. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  23. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  24. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  25. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  26. 2019Towards VQA Models That Can Readcited by 1,696
  27. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  28. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  29. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  30. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  31. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  32. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  33. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  34. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  35. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  36. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  37. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  38. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134

50 most cited of 369 All 369 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 4.

35 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  3. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lenscited by 42
  4. 2025A Survey on Human Interaction Motion Generationcited by 25
  5. 2025AnyMAC: Cascading Flexible Multi-Agent Collaboration via Next-Agent Predictioncited by 25
  6. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generationcited by 15
  7. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  8. 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospectscited by 12
  9. 2025IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarioscited by 12
  10. 2025Cloud-Based AI Systems: Leveraging Large Language Models for Intelligent Fault Detection and Autonomous Self-Healingcited by 10
  11. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  12. 2025Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?cited by 9
23 more
  1. 2025CoEmoGen: Towards Semantically-Coherent and Scalable Emotional Image Content Generationcited by 9
  2. 2025Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterizationcited by 8
  3. 2025Fundamentals of Building Autonomous LLM Agentscited by 7
  4. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  5. 2025A Grounded Memory System For Smart Personal Assistantscited by 5
  6. 2025FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoningcited by 5
  7. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5
  8. 20253DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Modelscited by 5
  9. 2025IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigationcited by 5
  10. 2025DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenescited by 4
  11. 2026ViT-AdaLA: Adapting Vision Transformers with Linear Attentioncited by 4
  12. 2025Window Token Concatenation for Efficient Visual Large Language Modelscited by 3
  13. 2025FaceInsight: A Multimodal Large Language Model for Face Perceptioncited by 2
  14. 2025ViT-Split: Unleashing the Power of Vision Foundation Models via Efficient Splitting Headscited by 2
  15. 2025A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospectscited by 2
  16. 2026Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraintscited by 2
  17. 2026Geometry-Guided Camera Motion Understanding in VideoLLMscited by 2
  18. 2025A Review on Large Language Models for Visual Analyticscited by 1
  19. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Modelscited by 1
  20. 2025Advancing Problem-Based Learning in Biomedical Engineering in the Era of Generative AIcited by 0
  21. 2025Leveraging Large Language Model for Intelligent Log Processing and Autonomous Debugging in Cloud AI Platformscited by 0
  22. 2025Galton's Law of Mediocrity: Why Large Language Models Regress to the Mean and Fail at Creativity in Advertisingcited by 0
  23. 2025Table Comprehension in Building Codes using Vision Language Models and Domain-Specific Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024An Introduction to Vision-Language Modeling
  2. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  3. 2024B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens
  4. 2024Rethinking VLMs and LLMs for Image Classification
  5. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  6. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  7. 2025Generalizing vision-language models to novel domains: A comprehensive survey
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks