# Fine-tuning Vision Transformers for the Prediction of State Variables in Ising Models

arXiv: https://arxiv.org/abs/2109.13925 · HTML: https://mlsanity.com/paper/2109.13925/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 28 Sep 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Deep Learning the Ising Model Near Criticality](https://mlsanity.com/paper/1708.04622/) (2017)
- [Learning the Ising Model with Generative Neural Networks](https://mlsanity.com/paper/2001.05361/) (2020)
- [Watch and learn -- a generalized approach for transferrable learning in deep neural networks via physical principles](https://mlsanity.com/paper/2003.02647/) (2020)
- [Sample-Efficient L0-L2 Constrained Structure Learning of Sparse Ising Models](https://mlsanity.com/paper/2012.01744/) (2020)
- [Can a CNN trained on the Ising model detect the phase transition of the q-state Potts model?](https://mlsanity.com/paper/2104.03632/) (2021)

## Built on by (most cited first, 3 of 3)

- [Fine-tuning Image Transformers using Learnable Memory](https://mlsanity.com/paper/2203.15243/) (2022)
- [A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques](https://mlsanity.com/paper/2302.08571/) (2023)
- [Emojich -- zero-shot emoji generation using Russian language: a technical report](https://mlsanity.com/paper/2112.02448/) (2021)

## Abstract (arXiv)

Transformers are state-of-the-art deep learning models that are composed of stacked attention and point-wise, fully connected layers designed for handling sequential data. Transformers are not only ubiquitous throughout Natural Language Processing (NLP), but, recently, they have inspired a new wave of Computer Vision (CV) applications research. In this work, a Vision Transformer (ViT) is applied to predict the state variables of 2-dimensional Ising model simulations. Our experiments show that ViT outperform state-of-the-art Convolutional Neural Networks (CNN) when using a small number of microstate images from the Ising model corresponding to various boundary conditions and temperatures. This work opens the possibility of applying ViT to other simulations, and raises interesting research directions on how attention maps can learn about the underlying physics governing different phenomena.