mlsanityData to 7 Oct 2026

Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models

arXiv 2311.06783 ↗First posted 12 Nov 2023cs.CV cs.MMcited by 152

Not on the map. Why?

Cited by 55 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multi-modality foundation models, as represented by GPT-4V, have brought a new paradigm for low-level visual perception and understanding tasks, that can respond to a broad range of natural human instructions in a model. While existing foundation models have shown exciting potentials on low-level visual tasks, their related abilities are still preliminary and need to be improved. In order to enhance these models, we conduct a large-scale subjective experiment collecting a vast number of real human feedbacks on low-level vision. Each feedback follows a pathway that starts with a detailed description on the low-level visual appearance (*e.g. clarity, color, brightness* of an image, and ends with an overall conclusion, with an average length of 45 words. The constructed **Q-Pathway** dataset includes 58K detailed human feedbacks on 18,973 images with diverse low-level appearance. Moreover, to enable foundation models to robustly respond to diverse types of questions, we design a GPT-participated conversion to process these feedbacks into diverse-format 200K instruction-response pairs. Experimental results indicate that the **Q-Instruct** consistently elevates low-level perception and understanding abilities across several foundational models. We anticipate that our datasets can pave the way for a future that general intelligence can perceive, understand low-level visual appearance and evaluate visual quality like a human. Our dataset, model zoo, and demo is published at: https://q-future.github.io/Q-Instruct.

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). Builds on 40 of its 63 references that are in the map. First posted on arXiv 12 Nov 2023.

Builds on

40 of 63 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
28 more
  1. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
  2. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  4. 2020Image Quality Assessment: Unifying Structure and Texture Similaritycited by 792
  5. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  9. 2018nocaps: novel object captioning at scalecited by 444
  10. 2019KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessmentcited by 351
  11. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
  12. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 314
  13. 2019Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Networkcited by 301
  14. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  15. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 276
  16. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  17. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  18. 2019From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Qualitycited by 221
  19. 2023TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessmentcited by 220
  20. 2015Massive Online Crowdsourced Study of Subjective and Objective Picture Qualitycited by 208
  21. 2022FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Samplingcited by 179
  22. 2023AGIQA-3K: An Open Database for AI-Generated Image Quality Assessmentcited by 142
  23. 2016Photo Aesthetics Ranking Network with Attributes and Content Adaptationcited by 122
  24. 2023VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretrainingcited by 63
  25. 2023Advancing Zero-Shot Digital Human Quality Assessment through Text-Prompted Evaluationcited by 27
  26. 2023Exploring Opinion-unaware Video Quality Assessment with Semantic Affinity Criterioncited by 23
  27. 2023Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Imagescited by 21
  28. 2023Towards Robust Text-Prompted Semantic Criterion for In-the-Wild Video Quality Assessmentcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 45; 2025: 62; 2026: 39.

152 papers.

Sort:

By area:

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  2. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  3. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  4. 2025Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distributioncited by 139
  5. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  6. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  7. 2023Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Modelscited by 104
  8. 2025Q-Insight: Understanding Image Quality via Visual Reinforcement Learningcited by 93
  9. 2024Towards Open-ended Visual Quality Comparisoncited by 85
  10. 2025VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rankcited by 77
  11. 2024Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Comparecited by 66
  12. 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 63
38 more
  1. 2024A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trendscited by 58
  2. 2024AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perceptioncited by 53
  3. 2024A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessmentcited by 43
  4. 2024An Intelligent Agentic System for Complex Image Restoration Problemscited by 41
  5. 2024Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessmentcited by 39
  6. 2024UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmarkcited by 39
  7. 2024A-Bench: Are LMMs Masters at Evaluating AI-generated Images?cited by 39
  8. 2024Enhancing Descriptive Image Quality Assessment with A Large-scale Multi-modal Datasetcited by 38
  9. 2024FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Modelscited by 36
  10. 2025ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understandingcited by 36
  11. 2025JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restorationcited by 32
  12. 2024EmoLLM: Multimodal Emotional Understanding Meets Large Language Modelscited by 31
  13. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  14. 2023Enhancing Diffusion Models with Text-Encoder Reinforcement Learningcited by 30
  15. 2024MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Modelcited by 29
  16. 2024LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Modelscited by 28
  17. 2024FineVQ: Fine-Grained User Generated Content Video Quality Assessmentcited by 28
  18. 2024Quality Assessment in the Era of Large Models: A Surveycited by 26
  19. 2025VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learningcited by 26
  20. 2024Large Multi-modality Model Assisted AI-Generated Image Quality Assessmentcited by 24
  21. 2023Image Super-Resolution with Text Prompt Diffusioncited by 22
  22. 2023Exploring the Naturalness of AI-Generated Imagescited by 22
  23. 2023Q-Boost: On Visual Quality Assessment Ability of Low-level Multi-Modality Foundation Modelscited by 22
  24. 2024Q-Refine: A Perceptual Quality Refiner for AI-Generated Imagecited by 21
  25. 20242AFC Prompting of Large Multimodal Models for Image Quality Assessmentcited by 20
  26. 2024NTIRE 2024 Quality Assessment of AI-Generated Content Challengecited by 20
  27. 2024LMM-PCQA: Assisting Point Cloud Quality Assessment with LMMcited by 19
  28. 2024Enhancing Blind Video Quality Assessment with Rich Quality-aware Featurescited by 19
  29. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  30. 2025UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texturecited by 19
  31. 2024VersaT2I: Improving Text-to-Image Models with Versatile Rewardcited by 17
  32. 2025HumanAesExpert: Advancing a Multi-Modality Foundation Model for Human Image Aesthetic Assessmentcited by 17
  33. 2024VQA2: Visual Question Answering for Video Quality Assessmentcited by 15
  34. 2024Video Quality Assessment: A Comprehensive Surveycited by 15
  35. 2025NTIRE 2025 Challenge on Short-form UGC Video Quality Assessment and Enhancement: Methods and Resultscited by 14
  36. 2024SEAGULL: No-reference Image Quality Assessment for Regions of Interest via Vision-Language Instruction Tuningcited by 13
  37. 2025Q-SiT: Teaching LMMs for Image Quality Scoring and Interpretingcited by 13
  38. 2024Grounding-IQA: Grounding Multimodal Language Model for Image Quality Assessmentcited by 12

50 most cited of 152 All 152 (JSON)