# Beyond Semantics: Modeling Factual and Affective Perceptual Experiences from Vision-Language Data

arXiv: https://arxiv.org/abs/2606.03345 · HTML: https://mlsanity.com/paper/2606.03345/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [Top2Vec: Distributed Representations of Topics](https://mlsanity.com/paper/2008.09470/) (2020)
- [Benchmarking Vision Language Models for Cultural Understanding](https://mlsanity.com/paper/2407.10920/) (2024)
- [Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!](https://mlsanity.com/paper/2004.14914/) (2020)
- [Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study](https://mlsanity.com/paper/2502.02481/) (2025)
- [It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collection](https://mlsanity.com/paper/2204.07660/) (2022)
- [Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations](https://mlsanity.com/paper/2202.04582/) (2022)
- [Affection: Learning Affective Explanations for Real-World Visual Data](https://mlsanity.com/paper/2210.01946/) (2022)
- [No Culture Left Behind: ArtELingo-28, a Benchmark of WikiArt with Captions in 28 Languages](https://mlsanity.com/paper/2411.03769/) (2024)
- [From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models](https://mlsanity.com/paper/2407.00263/) (2024)
- [CLIP-QDA: An Explainable Concept Bottleneck Model](https://mlsanity.com/paper/2312.00110/) (2023)
- [Write a Classifier: Predicting Visual Classifiers from Unstructured Text](https://mlsanity.com/paper/1601.00025/) (2015)
- [GPTopic: Dynamic and Interactive Topic Representations](https://mlsanity.com/paper/2403.03628/) (2024)
- [More than Memes: A Multimodal Topic Modeling Approach to Conspiracy Theories on Telegram](https://mlsanity.com/paper/2410.08642/) (2024)
- [Multilingual and Multimodal Topic Modelling with Pretrained Embeddings](https://mlsanity.com/paper/2211.08057/) (2022)
- [CEMTM: Contextual Embedding-based Multimodal Topic Modeling](https://mlsanity.com/paper/2509.11465/) (2025)

## Abstract (arXiv)

We present P-Topics (Perception Topics) modeling, a novel problem for understanding how images are perceived affectively and across cultures. The goal is to (1) discover and model the different perception experiences in a dataset of images and captions, where each experience is defined by an objective factual and a subjective affective aspect, and (2) associate images to their relevant perception experiences. We introduce **PercepT** (**Percep**tion topic **T**ransformer), a two-stage architecture that tackles P-Topics modeling. In the formation stage, percepT discovers *P-Topics* as visual-textual clusters using an unsupervised training objective, and dynamically selects the number of clusters to match the perceptual richness of the dataset. In the mapping stage, it learns *P-Topic mapping functions* via attention pooling to associate images to their respective clusters. On ArtELingo, PercepT achieves a silhouette score of **0.97** compared to **0.37** from the closest baseline reflecting better perceptual clusters. PercepT also achieves an AUC score of **0.94** compared to **0.77** showing better mapping to perceptual clusters. Human evaluation confirms that PercepT captures semantically meaningful perception experiences and significantly outperforms existing methods. Our implementation will be made public.