# Comparison of Image Processing Models in Quark Gluon Jet Classification

arXiv: https://arxiv.org/abs/2602.00141 · HTML: https://mlsanity.com/paper/2602.00141/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [A guide to convolution arithmetic for deep learning](https://mlsanity.com/paper/1603.07285/) (2016)
- [Efficient Training of Visual Transformers with Small Datasets](https://mlsanity.com/paper/2106.03746/) (2021)
- [Particle Transformer for Jet Tagging](https://mlsanity.com/paper/2202.03772/) (2022)
- [Energy Flow Networks: Deep Sets for Particle Jets](https://mlsanity.com/paper/1810.05165/) (2018)
- [ParticleNet: Jet Tagging via Particle Clouds](https://mlsanity.com/paper/1902.08570/) (2019)
- [Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models](https://mlsanity.com/paper/2401.13537/) (2024)
- [Jet-Images -- Deep Learning Edition](https://mlsanity.com/paper/1511.05190/) (2015)
- [Lorentz-Equivariant Geometric Algebra Transformers for High-Energy Physics](https://mlsanity.com/paper/2405.14806/) (2024)
- [Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models](https://mlsanity.com/paper/2403.07066/) (2024)
- [Deep learning in color: towards automated quark/gluon jet discrimination](https://mlsanity.com/paper/1612.01551/) (2016)
- [Does Robustness on ImageNet Transfer to Downstream Tasks?](https://mlsanity.com/paper/2204.03934/) (2022)
- [Learning Symmetry-Independent Jet Representations via Jet-Based Joint Embedding Predictive Architecture](https://mlsanity.com/paper/2412.05333/) (2024)
- [Content-Adaptive Downsampling in Convolutional Neural Networks](https://mlsanity.com/paper/2305.09504/) (2023)
- [Vision Transformers for End-to-End Quark-Gluon Jet Classification from Calorimeter Images](https://mlsanity.com/paper/2506.14934/) (2025)

## Abstract (arXiv)

Quark-gluon discrimination provides a useful test case for studying how different machine-learning architectures learn the spatial structure of QCD radiation. In this work, we compare convolutional neural network (CNN), Vision Transformers (ViT), and hierarchical Swin Transformers using the same three-channel jet-image representation, consisting of charged-particle momentum, neutral-particle momentum, and charged-particle multiplicity from PYTHIA 8 jets. We study their performance for different training-set sizes and fine-tuning configurations, with particular attention to the role of local and global information in the jet images. CNN and Swin models consistently perform better than ViT in the cases studied. Since both CNN and Swin retain a strong local component in their architectures, this suggests that local jet substructure plays an important role in quark-gluon discrimination. The performance of the hierarchical Swin model also suggests that combining local features over larger spatial scales is useful. Block-wise fine-tuning improves the performance of the Transformer models, although the improvement becomes smaller and the training less stable as more blocks are unfrozen. We also find that self-supervised Momentum Contrast (MoCo) pretraining improves the model initialization, particularly when the amount of labeled training data is limited. Based on these observations, we developed a smaller Swin model adopted to the jet-image representation used in this study. It achieves comparable performance with substantially fewer parameters. The results show that it is important to adapt the model architecture and training procedure to the specific input characteristics of High Energy Physics (HEP) data when applying vision models in HEP.