# DesignCLIP: Multimodal Learning with CLIP for Design Patent Understanding

arXiv: https://arxiv.org/abs/2508.15297 · HTML: https://mlsanity.com/paper/2508.15297/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [RoboCLIP: One Demonstration is Enough to Learn Robot Policies](https://mlsanity.com/paper/2310.07899/) (2023)
- [Extending CLIP for Category-to-image Retrieval in E-commerce](https://mlsanity.com/paper/2112.11294/) (2021)
- [CLIP-Lung: Textual Knowledge-Guided Lung Nodule Malignancy Prediction](https://mlsanity.com/paper/2304.08013/) (2023)
- [Contrastive Language, Action, and State Pre-training for Robot Learning](https://mlsanity.com/paper/2304.10782/) (2023)
- [A Survey on Patent Analysis: From NLP to Multimodal AI](https://mlsanity.com/paper/2404.08668/) (2024)
- [Enhancing Patent Retrieval using Text and Knowledge Graph Embeddings: A Technical Note](https://mlsanity.com/paper/2211.01976/) (2022)
- [The Role of Local Alignment and Uniformity in Image-Text Contrastive Learning on Medical Images](https://mlsanity.com/paper/2211.07254/) (2022)
- [Classification of Visualization Types and Perspectives in Patents](https://mlsanity.com/paper/2307.10471/) (2023)
- [CLIP feature-based randomized control using images and text for multiple tasks and robots](https://mlsanity.com/paper/2401.10085/) (2024)
- [Large Language Model Informed Patent Image Retrieval](https://mlsanity.com/paper/2404.19360/) (2024)

## Abstract (arXiv)

In the field of design patent analysis, traditional tasks such as patent classification and patent image retrieval heavily depend on the image data. However, patent images -- typically consisting of sketches with abstract and structural elements of an invention -- often fall short in conveying comprehensive visual context and semantic information. This inadequacy can lead to ambiguities in evaluation during prior art searches. Recent advancements in vision-language models, such as CLIP, offer promising opportunities for more reliable and accurate AI-driven patent analysis. In this work, we leverage CLIP models to develop a unified framework DesignCLIP for design patent applications with a large-scale dataset of U.S. design patents. To address the unique characteristics of patent data, DesignCLIP incorporates class-aware classification and contrastive learning, utilizing generated detailed captions for patent images and multi-views image learning. We validate the effectiveness of DesignCLIP across various downstream tasks, including patent classification and patent retrieval. Additionally, we explore multimodal patent retrieval, which provides the potential to enhance creativity and innovation in design by offering more diverse sources of inspiration. Our experiments show that DesignCLIP consistently outperforms baseline and SOTA models in the patent domain on all tasks. Our findings underscore the promise of multimodal approaches in advancing patent analysis. The codebase is available here: https://anonymous.4open.science/r/PATENTCLIP-4661/README.md.