# Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment

arXiv: https://arxiv.org/abs/2504.02522 · HTML: https://mlsanity.com/paper/2504.02522/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment](https://mlsanity.com/paper/1910.06180/) (2019)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://mlsanity.com/paper/2307.06304/) (2023)
- [TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment](https://mlsanity.com/paper/2308.03060/) (2023)
- [FlexiViT: One Model for All Patch Sizes](https://mlsanity.com/paper/2212.08013/) (2022)
- [Photo Aesthetics Ranking Network with Attributes and Content Adaptation](https://mlsanity.com/paper/1606.01621/) (2016)
- [Token Fusion: Bridging the Gap between Token Pruning and Token Merging](https://mlsanity.com/paper/2312.01026/) (2023)
- [Any-resolution Training for High-resolution Image Synthesis](https://mlsanity.com/paper/2204.07156/) (2022)
- [VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining](https://mlsanity.com/paper/2303.14302/) (2023)
- [Personalized Image Aesthetics Assessment with Rich Attributes](https://mlsanity.com/paper/2203.16754/) (2022)
- [Effective Aesthetics Prediction with Multi-level Spatially Pooled Features](https://mlsanity.com/paper/1904.01382/) (2019)
- [Towards Artistic Image Aesthetics Assessment: a Large-scale Dataset and a New Method](https://mlsanity.com/paper/2303.15166/) (2023)
- [Learning to Resize Images for Computer Vision Tasks](https://mlsanity.com/paper/2103.09950/) (2021)
- [Vision Transformers with Mixed-Resolution Tokenization](https://mlsanity.com/paper/2304.00287/) (2023)
- [Adaptive Fractional Dilated Convolution Network for Image Aesthetics Assessment](https://mlsanity.com/paper/2004.03015/) (2020)
- [MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers](https://mlsanity.com/paper/2307.02321/) (2023)
- [AiluRus: A Scalable ViT Framework for Dense Prediction](https://mlsanity.com/paper/2311.01197/) (2023)
- [KonX: Cross-Resolution Image Quality Assessment](https://mlsanity.com/paper/2212.05813/) (2022)
- [Image Aesthetics Assessment Using Graph Attention Network](https://mlsanity.com/paper/2206.12869/) (2022)
- [MRET: Multi-resolution Transformer for Video Quality Assessment](https://mlsanity.com/paper/2303.07489/) (2023)

## Built on by (most cited first, 5 of 5)

- [Fine-grained Image Aesthetic Assessment: Learning Discriminative Scores from Relative Ranks](https://mlsanity.com/paper/2603.03907/) (2026)
- [Beyond Visual Similarity: Rule-Guided Multimodal Clustering with explicit domain rules](https://mlsanity.com/paper/2509.20501/) (2025)
- [From Concepts to Judgments: Interpretable Image Aesthetic Assessment](https://mlsanity.com/paper/2603.18108/) (2026)
- [Aes3D: Aesthetic Assessment in 3D Gaussian Splatting](https://mlsanity.com/paper/2605.05155/) (2026)
- [XPASS-Vis: A Dataset for Cross-Domain Personalized Image Aesthetic Assessment](https://mlsanity.com/paper/2606.15629/) (2026)

## Abstract (arXiv)

The capacity of Vision transformers (ViTs) to handle variable-sized inputs is often constrained by computational complexity and batch processing limitations. Consequently, ViTs are typically trained on small, fixed-size images obtained through downscaling or cropping. While reducing computational burden, these methods result in significant information loss, negatively affecting tasks like image aesthetic assessment. We introduce Charm, a novel tokenization approach that preserves Composition, High-resolution, Aspect Ratio, and Multi-scale information simultaneously. Charm prioritizes high-resolution details in specific regions while downscaling others, enabling shorter fixed-size input sequences for ViTs while incorporating essential information. Charm is designed to be compatible with pre-trained ViTs and their learned positional embeddings. By providing multiscale input and introducing variety to input tokens, Charm improves ViT performance and generalizability for image aesthetic assessment. We avoid cropping or changing the aspect ratio to further preserve information. Extensive experiments demonstrate significant performance improvements on various image aesthetic and quality assessment datasets (up to 8.1 %) using a lightweight ViT backbone. Code and pre-trained models are available at https://github.com/FBehrad/Charm.