mlsanityData to 7 Oct 2026

SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant

arXiv 2403.11299 ↗First posted 17 Mar 2024cs.CV cs.AI cs.CLcited by 18

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent advances in vision-language models have shown notable generalization in broad tasks through visual instruction tuning. However, bridging the gap between the pre-trained vision encoder and the large language models (LLMs) becomes the whole network's bottleneck. To improve cross-modality alignment, existing works usually consider more visual instruction data covering a broader range of vision tasks to fine-tune the model for question-answering, which, however, is costly to obtain and has not thoroughly explored the rich contextual information contained in images. This paper first attempts to harness the overlooked context within visual instruction data, training the model to self-supervised "learning" how to ask high-quality questions. In this way, we introduce a novel framework named SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant. SQ-LLaVA exhibits proficiency in generating flexible and meaningful image-related questions while analyzing the visual clue and prior language knowledge, signifying an advanced level of generalized visual understanding. Moreover, fine-tuning SQ-LLaVA on higher-quality instruction data shows a performance improvement compared with traditional visual-instruction tuning methods. This improvement highlights the efficacy of self-questioning techniques in achieving a deeper and more nuanced comprehension of visual content across various contexts.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 59 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

43 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
29 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  13. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  14. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  15. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  16. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  17. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  18. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  19. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  20. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  21. 2018Object Hallucination in Image Captioningcited by 661
  22. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  23. 2018nocaps: novel object captioning at scalecited by 444
  24. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  25. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  26. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  27. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  28. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  29. 2023Cross-Domain Image Captioning with Discriminative Finetuningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 9; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answeringcited by 21
  2. 2024Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Promptscited by 15
  3. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  4. 2024VideoSAVi: Self-Aligned Video Language Models without Human Supervisioncited by 7
  5. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  6. 2025Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networkscited by 5
  7. 2024Transparent and Coherent Procedural Mistake Detectioncited by 4
  8. 2024VORD: Visual Ordinal Calibration for Mitigating Object Hallucinations in Large Vision-Language Modelscited by 4
  9. 2025SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winnercited by 2
  10. 2025Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learningcited by 2
  11. 2025X-CoT: Explainable Text-to-Video Retrieval via LLM-based Chain-of-Thought Reasoningcited by 2
  12. 2025Visual Self-Refinement for Autoregressive Modelscited by 2
6 more
  1. 2025A Fully Automated Pipeline for Conversational Discourse Annotation: Tree Scheme Generation and Labeling with Large Language Modelscited by 1
  2. 2025HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Modelscited by 1
  3. 2024RaCT: Ranking-aware Chain-of-Thought Optimization for LLMscited by 0
  4. 2026The Detector Teaches Itself: Lightweight Self-Supervised Adaptation for Open-Vocabulary Object Detectioncited by 0
  5. 2026Personalized Object Identification and Localization via In-Context Inference with Vision-Language Modelscited by 0
  6. 2026Information-Regularized Attention for Visual-Centric Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMs
  2. 2025Instruction-tuned Self-Questioning Framework for Multimodal Reasoning
  3. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
  4. 2025Towards Self-Refinement of Vision-Language Models with Triangular Consistency
  5. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehension
  6. 2024SADL: An Effective In-Context Learning Method for Compositional Visual QA
  7. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Model
  8. 2025MC-LLaVA: Multi-Concept Personalized Vision-Language Model