mlsanityData to 7 Oct 2026

Target Prompting for Information Extraction with Vision Language Model

arXiv 2408.03834 ↗First posted 7 Aug 2024cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The recent trend in the Large Vision and Language model has brought a new change in how information extraction systems are built. VLMs have set a new benchmark with their State-of-the-art techniques in understanding documents and building question-answering systems across various industries. They are significantly better at generating text from document images and providing accurate answers to questions. However, there are still some challenges in effectively utilizing these models to build a precise conversational system. General prompting techniques used with large language models are often not suitable for these specially designed vision language models. The output generated by such generic input prompts is ordinary and may contain information gaps when compared with the actual content of the document. To obtain more accurate and specific answers, a well-targeted prompt is required by the vision language model, along with the document image. In this paper, a technique is discussed called Target prompting, which focuses on explicitly targeting parts of document images and generating related answers from those specific regions only. The paper also covers the evaluation of response for each prompting technique using different user queries and input prompts.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 6 of its 20 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

6 of 20 references.

References · 6

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  3. 2023A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challengecited by 55
  4. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  5. 2023TAP: Targeted Prompting for Task Adaptive Generation of Textual Training Instances for Visual Classificationcited by 8
  6. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extractioncited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
  2. 2023Language Models as Black-Box Optimizers for Vision-Language Models
  3. 2024Targeted Visual Prompting for Medical Visual Question Answering
  4. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  5. 2024Attention Prompting on Image for Large Vision-Language Models
  6. 2024Right this way: Can VLMs Guide Us to See More to Answer Questions?
  7. 2025Joint Extraction Matters: Prompt-Based Visual Question Answering for Multi-Field Document Information Extraction
  8. 2025Modeling Variants of Prompts for Vision-Language Models