# Sampling two-dimensional spin systems with transformers

arXiv: https://arxiv.org/abs/2604.27738 · HTML: https://mlsanity.com/paper/2604.27738/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [MADE: Masked Autoencoder for Distribution Estimation](https://mlsanity.com/paper/1502.03509/) (2015)
- [Flow-based generative models for Markov chain Monte Carlo in lattice field theory](https://mlsanity.com/paper/1904.12072/) (2019)
- [Solving Statistical Mechanics Using Variational Autoregressive Networks](https://mlsanity.com/paper/1809.10606/) (2018)
- [Normalizing flows for lattice gauge theory in arbitrary space-time dimension](https://mlsanity.com/paper/2305.02402/) (2023)
- [Hierarchical autoregressive neural networks for statistical systems](https://mlsanity.com/paper/2203.10989/) (2022)
- [The autoregressive neural network architecture of the Boltzmann distribution of pairwise interacting spins systems](https://mlsanity.com/paper/2302.08347/) (2023)
- [Multilevel Generative Samplers for Investigating Critical Phenomena](https://mlsanity.com/paper/2503.08918/) (2025)
- [Simulating first-order phase transition with hierarchical autoregressive networks](https://mlsanity.com/paper/2212.04955/) (2022)
- [Mutual information of spin systems from autoregressive neural networks](https://mlsanity.com/paper/2304.13412/) (2023)
- [Hierarchical autoregressive neural networks in three-dimensional statistical system](https://mlsanity.com/paper/2503.08610/) (2025)
- [Estimation of the reduced density matrix and entanglement entropies using autoregressive networks](https://mlsanity.com/paper/2506.04170/) (2025)

## Built on by (most cited first, 2 of 2)

- [Variational Autoregressive Networks with probability priors](https://mlsanity.com/paper/2605.16020/) (2026)
- [Fixed-point neural samplers on discrete spaces](https://mlsanity.com/paper/2610.01739/) (2026)

## Abstract (arXiv)

Autoregressive Neural Networks based on dense or convolutional layers have recently been shown to be a viable strategy for generating classical spin systems. Unlike these methods, sampling with transformers is commonly considered to be computationally inefficient. In this work, we propose a novel approach to transformer-based neural samplers in which we generate not a single spin per step but groups of spins. As an additional improvement, we construct a model of approximated probabilities, further improving the efficiency of the algorithm. Despite our approach being computationally heavier than dense networks or CNN-based approaches, we were able to sample larger systems of up to 180 × 180 spins in case of the Ising model. The Effective Sample Size of our sampler is ~ 20 times larger than that of the previous state-of-the-art neural sampler when trained for the 128 × 128 Ising model at critical temperature. Finally, we also test our algorithm on the 2D Edwards-Anderson model, where we train 64× 64 spin systems.