# Describing Sets of Images with Textual-PCA

arXiv: https://arxiv.org/abs/2210.12112 · HTML: https://mlsanity.com/paper/2210.12112/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [Explaining Classifiers with Causal Concept Effect (CaCE)](https://mlsanity.com/paper/1907.07165/) (2019)
- [Explaining in Style: Training a GAN to explain a classifier in StyleSpace](https://mlsanity.com/paper/2104.13369/) (2021)
- [Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)](https://mlsanity.com/paper/1412.6632/) (2014)
- [Language Models Can See: Plugging Visual Controls in Text Generation](https://mlsanity.com/paper/2205.02655/) (2022)
- [A Simple Baseline for Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1904.05876/) (2019)
- [From Facial Parts Responses to Face Detection: A Deep Learning Approach](https://mlsanity.com/paper/1509.06451/) (2015)
- [Zero-Shot Video Captioning with Evolving Pseudo-Tokens](https://mlsanity.com/paper/2207.11100/) (2022)
- [Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation](https://mlsanity.com/paper/1411.7399/) (2014)
- [Latent Space Explanation by Intervention](https://mlsanity.com/paper/2112.04895/) (2021)
- [Ordered Attention for Coherent Visual Storytelling](https://mlsanity.com/paper/2108.02180/) (2021)

## Built on by (most cited first, 1 of 1)

- [Effective Human-AI Teams via Learned Natural Language Rules and Onboarding](https://mlsanity.com/paper/2311.01007/) (2023)

## Abstract (arXiv)

We seek to semantically describe a set of images, capturing both the attributes of single images and the variations within the set. Our procedure is analogous to Principle Component Analysis, in which the role of projection vectors is replaced with generated phrases. First, a centroid phrase that has the largest average semantic similarity to the images in the set is generated, where both the computation of the similarity and the generation are based on pretrained vision-language models. Then, the phrase that generates the highest variation among the similarity scores is generated, using the same models. The next phrase maximizes the variance subject to being orthogonal, in the latent space, to the highest-variance phrase, and the process continues. Our experiments show that our method is able to convincingly capture the essence of image sets and describe the individual elements in a semantically meaningful way within the context of the entire set. Our code is available at: https://github.com/OdedH/textual-pca.