# PatFig: Generating Short and Long Captions for Patent Figures

arXiv: https://arxiv.org/abs/2309.08379 · HTML: https://mlsanity.com/paper/2309.08379/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 15 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [Vision-Language Models in Remote Sensing: Current Progress and Future Trends](https://mlsanity.com/paper/2305.05726/) (2023)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2008.06239/) (2020)
- [Figure Captioning with Reasoning and Sequence-Level Training](https://mlsanity.com/paper/1906.02850/) (2019)
- [DocTr: Document Transformer for Structured Information Extraction in Documents](https://mlsanity.com/paper/2307.07929/) (2023)
- [Abstract Images Have Different Levels of Retrievability Per Reverse Image Search Engine](https://mlsanity.com/paper/2211.02115/) (2022)

## Built on by (most cited first, 12 of 12)

- [GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions](https://mlsanity.com/paper/2310.15405/) (2023)
- [Natural Language Processing in the Patent Domain: A Survey](https://mlsanity.com/paper/2403.04105/) (2024)
- [Can Large Language Models Generate High-quality Patent Claims?](https://mlsanity.com/paper/2406.19465/) (2024)
- [PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figures](https://mlsanity.com/paper/2501.15074/) (2025)
- [Patent-CR: A Dataset for Patent Claim Revision](https://mlsanity.com/paper/2412.02549/) (2024)
- [DocKD: Knowledge Distillation from LLMs for Open-World Document Understanding Models](https://mlsanity.com/paper/2410.03061/) (2024)
- [AutoSpec: An Agentic Framework for Automatically Drafting Patent Specification](https://mlsanity.com/paper/2509.19640/) (2025)
- [Exploiting LMM-based knowledge for image classification tasks](https://mlsanity.com/paper/2406.03071/) (2024)
- [SkinGEN: an Explainable Dermatology Diagnosis-to-Generation Framework with Interactive Vision-Language Models](https://mlsanity.com/paper/2404.14755/) (2024)
- [PatentVision: A multimodal method for drafting patent applications](https://mlsanity.com/paper/2510.09762/) (2025)
- [Hierarchical Multi-Positive Contrastive Learning for Patent Image Retrieval](https://mlsanity.com/paper/2506.13496/) (2025)
- [Patentformer: A demonstration of AI-assisted automated patent drafting](https://mlsanity.com/paper/2510.09752/) (2025)

## Abstract (arXiv)

This paper introduces Qatent PatFig, a novel large-scale patent figure dataset comprising 30,000+ patent figures from over 11,000 European patent applications. For each figure, this dataset provides short and long captions, reference numerals, their corresponding terms, and the minimal claim set that describes the interactions between the components of the image. To assess the usability of the dataset, we finetune an LVLM model on Qatent PatFig to generate short and long descriptions, and we investigate the effects of incorporating various text-based cues at the prediction stage of the patent figure captioning process.