# For a semiotic AI: Bridging computer vision and visual semiotics for computational observation of large scale facial image archives

arXiv: https://arxiv.org/abs/2407.03268 · HTML: https://mlsanity.com/paper/2407.03268/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 98 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 30 of 35)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [Exploring the Limits of Out-of-Distribution Detection](https://mlsanity.com/paper/2106.03004/) (2021)
- [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://mlsanity.com/paper/1706.08500/) (2017)
- [General Facial Representation Learning in a Visual-Linguistic Manner](https://mlsanity.com/paper/2112.03109/) (2021)
- [PifPaf: Composite Fields for Human Pose Estimation](https://mlsanity.com/paper/1903.06593/) (2019)
- [Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation](https://mlsanity.com/paper/2109.09881/) (2021)
- [Dense Extreme Inception Network: Towards a Robust CNN Model for Edge Detection](https://mlsanity.com/paper/1909.01955/) (2019)
- [Automatic Understanding of Image and Video Advertisements](https://mlsanity.com/paper/1707.03067/) (2017)
- [6D Rotation Representation For Unconstrained Head Pose Estimation](https://mlsanity.com/paper/2202.12555/) (2022)
- [Convolutional Character Networks](https://mlsanity.com/paper/1910.07954/) (2019)
- [Towards Artistic Image Aesthetics Assessment: a Large-scale Dataset and a New Method](https://mlsanity.com/paper/2303.15166/) (2023)
- [A Step Toward More Inclusive People Annotations for Fairness](https://mlsanity.com/paper/2105.02317/) (2021)
- [Attribute-preserving Face Dataset Anonymization via Latent Code Optimization](https://mlsanity.com/paper/2303.11296/) (2023)
- [The Shape of Art History in the Eyes of the Machine](https://mlsanity.com/paper/1801.07729/) (2018)
- [ADVISE: Symbolism and External Knowledge for Decoding Advertisements](https://mlsanity.com/paper/1711.06666/) (2017)
- [One-Shot Unsupervised Cross-Domain Detection](https://mlsanity.com/paper/2005.11610/) (2020)
- [3DGazeNet: Generalizing Gaze Estimation with Weak-Supervision from Synthetic Views](https://mlsanity.com/paper/2212.02997/) (2022)
- [The Visual Social Distancing Problem](https://mlsanity.com/paper/2005.04813/) (2020)
- [Understanding Compositional Structures in Art Historical Images using Pose and Gaze Priors](https://mlsanity.com/paper/2009.03807/) (2020)
- [What your Facebook Profile Picture Reveals about your Personality](https://mlsanity.com/paper/1708.01292/) (2017)
- [Visual DNA: Representing and Comparing Images using Distributions of Neuron Activations](https://mlsanity.com/paper/2304.10036/) (2023)
- [Latent Diffusion Models for Attribute-Preserving Image Anonymization](https://mlsanity.com/paper/2403.14790/) (2024)
- [Contextualizing Internet Memes Across Social Media Platforms](https://mlsanity.com/paper/2311.11157/) (2023)

## Built on by (most cited first, 1 of 1)

- [On Semiotic-Grounded Interpretive Evaluation of Generative Art](https://mlsanity.com/paper/2604.08641/) (2026)

## Abstract (arXiv)

Social networks are creating a digital world in which the cognitive, emotional, and pragmatic value of the imagery of human faces and bodies is arguably changing. However, researchers in the digital humanities are often ill-equipped to study these phenomena at scale. This work presents FRESCO (Face Representation in E-Societies through Computational Observation), a framework designed to explore the socio-cultural implications of images on social media platforms at scale. FRESCO deconstructs images into numerical and categorical variables using state-of-the-art computer vision techniques, aligning with the principles of visual semiotics. The framework analyzes images across three levels: the plastic level, encompassing fundamental visual features like lines and colors; the figurative level, representing specific entities or concepts; and the enunciation level, which focuses particularly on constructing the point of view of the spectator and observer. These levels are analyzed to discern deeper narrative layers within the imagery. Experimental validation confirms the reliability and utility of FRESCO, and we assess its consistency and precision across two public datasets. Subsequently, we introduce the FRESCO score, a metric derived from the framework's output that serves as a reliable measure of similarity in image content.