# IsingFormer: Augmenting Parallel Tempering With Learned Proposals

arXiv: https://arxiv.org/abs/2509.23043 · HTML: https://mlsanity.com/paper/2509.23043/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 62 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Flow-based generative models for Markov chain Monte Carlo in lattice field theory](https://mlsanity.com/paper/1904.12072/) (2019)
- [Combinatorial Optimization with Physics-Inspired Graph Neural Networks](https://mlsanity.com/paper/2107.01188/) (2021)
- [Equivariant flow-based sampling for lattice gauge theory](https://mlsanity.com/paper/2003.06413/) (2020)
- [Solving Statistical Mechanics Using Variational Autoregressive Networks](https://mlsanity.com/paper/1809.10606/) (2018)
- [Sampling using SU(N) gauge equivariant flows](https://mlsanity.com/paper/2008.05456/) (2020)
- [A Diffusion Model Framework for Unsupervised Neural Combinatorial Optimization](https://mlsanity.com/paper/2406.01661/) (2024)
- [Variational Neural Annealing](https://mlsanity.com/paper/2101.10154/) (2021)
- [Gauge-equivariant flow models for sampling in lattice field theories with pseudofermions](https://mlsanity.com/paper/2207.08945/) (2022)
- [Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics](https://mlsanity.com/paper/2502.08696/) (2025)
- [Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective](https://mlsanity.com/paper/2308.14085/) (2023)
- [Massively Parallel Probabilistic Computing with Sparse Ising Machines](https://mlsanity.com/paper/2110.02481/) (2021)
- [Accelerate Monte Carlo Simulations with Restricted Boltzmann Machines](https://mlsanity.com/paper/1610.02746/) (2016)
- [Unbiased Monte Carlo Cluster Updates with Autoregressive Neural Networks](https://mlsanity.com/paper/2105.05650/) (2021)
- [Accelerated Parallel Tempering via Neural Transports](https://mlsanity.com/paper/2502.10328/) (2025)
- [Learning Thermodynamics with Boltzmann Machines](https://mlsanity.com/paper/1606.02718/) (2016)
- [Controlled Online Optimization Learning (COOL): Finding the ground state of spin Hamiltonians with reinforcement learning](https://mlsanity.com/paper/2003.00011/) (2020)
- [Deep Learning the Ising Model Near Criticality](https://mlsanity.com/paper/1708.04622/) (2017)
- [All-to-all reconfigurability with sparse and higher-order Ising machines](https://mlsanity.com/paper/2312.08748/) (2023)
- [Nonequilibrium Monte Carlo for unfreezing variables in hard combinatorial optimization](https://mlsanity.com/paper/2111.13628/) (2021)
- [Message Passing Variational Autoregressive Network for Solving Intractable Ising Models](https://mlsanity.com/paper/2404.06225/) (2024)
- [Discrete generative diffusion models without stochastic differential equations: a tensor network approach](https://mlsanity.com/paper/2407.11133/) (2024)
- [A Generative Neural Annealer for Black-Box Combinatorial Optimization](https://mlsanity.com/paper/2505.09742/) (2025)

## Built on by (most cited first, 3 of 3)

- [Demonstrating Real Advantage of Machine-Learning-Enhanced Monte Carlo for Combinatorial Optimization](https://mlsanity.com/paper/2510.19544/) (2025)
- [From Independent to Correlated Diffusion: Generalized Generative Modeling with Probabilistic Computers](https://mlsanity.com/paper/2603.27996/) (2026)
- [Probabilistic Algorithms for Ising Machines from Optimization to Generative AI](https://mlsanity.com/paper/2610.03972/) (2026)

## Abstract (arXiv)

Generative models have been extensively used to accelerate MCMC mixing for sampling and optimization, but their effective integration with standard MCMC remains an open question. Here, we introduce a global proposal move in which finite-temperature configurations from an external generator are used as proposals within Parallel Tempering (PT). We examine a specific generator, IsingFormer, a Transformer trained on long-run MCMC configurations intended to approximate equilibrium distributions, and call the resulting framework Transformer-Augmented Parallel Tempering (TAPT). The IsingFormer exhibits two useful capabilities: interpolation to untrained β values and conditional completion under clamped settings absent from training. On 3D spin-glass instances, TAPT reaches substantially lower residual energies than standard PT in fewer Monte Carlo sweeps. On integer factorization, we show how a structured problem encoding can amortize the training cost by training IsingFormer once and reusing the same model across target products that were not imposed during training. Finally, in a scaling study that uses long-run MCMC configurations as proposals, with proposal generation excluded from the timing, TAPT reduces the fitted time-to-solution exponent by approximately 33% relative to PT over the tested problem sizes.