# Integrating Query-aware Segmentation and Cross-Attention for Robust VQA

arXiv: https://arxiv.org/abs/2407.12055 · HTML: https://mlsanity.com/paper/2407.12055/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 5 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Builds on (most cited first, 3 of 3)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)

## Abstract (arXiv)

This paper introduces a method for VizWiz-VQA using LVLM with trainable cross-attention and LoRA finetuning. We train the model with the following conditions: 1) Training with original images. 2) Training with enhanced images using CLIPSeg to highlight or contrast the original image. 3) Training with integrating the output features of Vision Transformer (ViT) and CLIPSeg features of the original images. Then, we ensemble the results based on Levenshtein distance to enhance the prediction of the final answer. In the experiments, we demonstrate and analyze the proposed method's effectiveness.