# Fragment-Aware Vision Transformers for Fresco-Fragment Style Classification

arXiv: https://arxiv.org/abs/2609.21012 · HTML: https://mlsanity.com/paper/2609.21012/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [GNNExplainer: Generating Explanations for Graph Neural Networks](https://mlsanity.com/paper/1903.03894/) (2019)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Parameterized Explainer for Graph Neural Network](https://mlsanity.com/paper/2011.04573/) (2020)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [Re-assembling the past: The RePAIR dataset and benchmark for real world 2D and 3D puzzle solving](https://mlsanity.com/paper/2410.24010/) (2024)
- [Hypergraph Vision Transformers: Images are More than Nodes, More than Edges](https://mlsanity.com/paper/2504.08710/) (2025)
- [A Unified and Biologically-Plausible Relational Graph Representation of Vision Transformers](https://mlsanity.com/paper/2206.11073/) (2022)
- [Recognizing Artistic Style of Archaeological Image Fragments Using Deep Style Extrapolation](https://mlsanity.com/paper/2501.00836/) (2025)
- [SAG-ViT: A Scale-Aware, High-Fidelity Patching Approach with Graph Attention for Vision Transformers](https://mlsanity.com/paper/2411.09420/) (2024)
- [The Cow of Rembrandt - Analyzing Artistic Prompt Interpretation in Text-to-Image Models](https://mlsanity.com/paper/2507.23313/) (2025)
- [ShapBPT: Image Feature Attributions Using Data-Aware Binary Partition Trees](https://mlsanity.com/paper/2602.07047/) (2026)
- [Semantic Graph Consistency: Going Beyond Patches for Regularizing Self-Supervised Vision Transformers](https://mlsanity.com/paper/2406.12944/) (2024)

## Abstract (arXiv)

Artistic style classification is usually studied on complete artworks, where models can exploit global composition, spatial organisation, and iconographic structure. In archaeological settings, however, artworks often survive only as fragmented remains, forcing recognition from incomplete, irregular, and context-limited visual evidence. We study fresco-fragment style classification using a progressive transformer-based framework. Starting from a ViT-B/16 baseline, we introduce foreground-guided masking to suppress background-only tokens, inpainting-based geometric regularisation to align irregular fragment supports with the ViT patch grid, and a supervised contrastive objective that operates on predictive distributions through a Kullback-Leibler similarity and consistently improves every branch. We combine the branches with a deliberately simple learnable logit ensemble. Experiments on CLEOPATRA and POMPAAF show that fragment-aware modelling improves over the standard ViT baseline, with the ensemble increasing accuracy from 0.604 to 0.656 and macro-F1 from 0.596 to 0.648 on CLEOPATRA, and outperforming the best single branch in four of six fragmentation settings on POMPAAF. We additionally evaluate a more complex graph-fusion variant and find that it matches the simple ensemble on POMPAAF while offering only a small, dataset-specific gain on CLEOPATRA, which does not justify its added complexity. Beyond these empirical gains, our contribution is twofold: a distribution-level contrastive objective that consistently sharpens single-branch recognition, and an interpretability analysis that verifies the models exploit genuine painted evidence, while quantifying that the inpainting-based branch draws part of its attribution from the synthesised surround.