mlsanityData to 14 Oct 2026

Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision

arXiv 2309.14181 ↗First posted 25 Sep 2023cs.CV cs.AI cs.MMcited by 249

Not on the map. Why?

Cited by 117 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rapid evolution of Multi-modality Large Language Models (MLLMs) has catalyzed a shift in computer vision from specialized models to general-purpose foundation models. Nevertheless, there is still an inadequacy in assessing the abilities of MLLMs on low-level visual perception and understanding. To address this gap, we present Q-Bench, a holistic benchmark crafted to systematically evaluate potential abilities of MLLMs on three realms: low-level visual perception, low-level visual description, and overall visual quality assessment. a) To evaluate the low-level perception ability, we construct the LLVisionQA dataset, consisting of 2,990 diverse-sourced images, each equipped with a human-asked question focusing on its low-level attributes. We then measure the correctness of MLLMs on answering these questions. b) To examine the description ability of MLLMs on low-level information, we propose the LLDescribe dataset consisting of long expert-labelled golden low-level text descriptions on 499 images, and a GPT-involved comparison pipeline between outputs of MLLMs and the golden descriptions. c) Besides these two tasks, we further measure their visual quality assessment ability to align with human opinion scores. Specifically, we design a softmax-based strategy that enables MLLMs to predict quantifiable quality scores, and evaluate them on various existing image quality assessment (IQA) datasets. Our evaluation across the three abilities confirms that MLLMs possess preliminary low-level visual skills. However, these skills are still unstable and relatively imprecise, indicating the need for specific enhancements on MLLMs towards these abilities. We hope that our benchmark can encourage the research community to delve deeper to discover and enhance these untapped potentials of MLLMs. Project Page: https://q-future.github.io/Q-Bench.

Key facts. Cited by 249 papers in the map (data through 14 Oct 2026). Builds on 38 of its 60 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

38 of 60 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
26 more
  1. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  3. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  4. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 991
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  7. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  10. 2018nocaps: novel object captioning at scalecited by 444
  11. 2017NIMA: Neural Image Assessmentcited by 389
  12. 2019KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessmentcited by 355
  13. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 336
  14. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 320
  15. 2019From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Qualitycited by 223
  16. 2015Massive Online Crowdsourced Study of Subjective and Objective Picture Qualitycited by 209
  17. 2022FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Samplingcited by 179
  18. 2020Patch-VQ: 'Patching Up' the Video Quality Problemcited by 147
  19. 2023AGIQA-3K: An Open Database for AI-Generated Image Quality Assessmentcited by 142
  20. 2016Photo Aesthetics Ranking Network with Attributes and Content Adaptationcited by 125
  21. 2022Neighbourhood Representative Sampling for Efficient End-to-end Video Quality Assessmentcited by 50
  22. 2023Advancing Zero-Shot Digital Human Quality Assessment through Text-Prompted Evaluationcited by 27
  23. 2023Exploring Opinion-unaware Video Quality Assessment with Semantic Affinity Criterioncited by 23
  24. 2023Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Imagescited by 21
  25. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Modelscited by 13
  26. 2022Exploring and Evaluating Image Restoration Potential in Dynamic Scenescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 98; 2025: 85; 2026: 56.

249 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  3. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  4. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  5. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  6. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
  7. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  8. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  9. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  10. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  11. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  12. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
38 more
  1. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  2. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  3. 2023Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Modelscited by 152
  4. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  5. 2025Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distributioncited by 141
  6. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  7. 2023Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Modelscited by 105
  8. 2025Q-Insight: Understanding Image Quality via Visual Reinforcement Learningcited by 98
  9. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 90
  10. 2024Towards Open-ended Visual Quality Comparisoncited by 85
  11. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  12. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 82
  13. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 79
  14. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  15. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 67
  16. 2024Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Comparecited by 66
  17. 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 65
  18. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 63
  19. 2024AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perceptioncited by 54
  20. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 54
  21. 2024Towards A Better Metric for Text-to-Video Generationcited by 51
  22. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  23. 2024A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessmentcited by 43
  24. 2024An Intelligent Agentic System for Complex Image Restoration Problemscited by 43
  25. 2024UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmarkcited by 42
  26. 2024XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolutioncited by 41
  27. 2024SHIELD : An Evaluation Benchmark for Face Spoofing and Forgery Detection with Multimodal Large Language Modelscited by 40
  28. 2024Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessmentcited by 39
  29. 2024A-Bench: Are LMMs Masters at Evaluating AI-generated Images?cited by 39
  30. 2024Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMscited by 39
  31. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 39
  32. 2024Enhancing Descriptive Image Quality Assessment with A Large-scale Multi-modal Datasetcited by 38
  33. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  34. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  35. 2024FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Modelscited by 36
  36. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 35
  37. 2024AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessmentcited by 33
  38. 2025AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Modelscited by 33

50 most cited of 249 All 249 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?
  2. 2024LM4LV: A Frozen Large Language Model for Low-level Vision Tasks
  3. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  4. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
  5. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
  6. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  7. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?
  8. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks