# PuzLM: Solving Jigsaw Puzzles with Sequence-to-Sequence Language Models

arXiv: https://arxiv.org/abs/2511.06315 · HTML: https://mlsanity.com/paper/2511.06315/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 57 references that are in the map.
- First posted on arXiv 9 Nov 2025.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [An Image is Worth 32 Tokens for Reconstruction and Generation](https://mlsanity.com/paper/2406.07550/) (2024)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL](https://mlsanity.com/paper/2205.06983/) (2022)
- [Iterative Reorganization with Weak Spatial Constraints: Solving Arbitrary Jigsaw Puzzles for Unsupervised Representation Learning](https://mlsanity.com/paper/1812.00329/) (2018)
- [The Met Dataset: Instance-level Recognition for Artworks](https://mlsanity.com/paper/2202.01747/) (2022)
- [Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzles](https://mlsanity.com/paper/2505.23590/) (2025)
- [DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly](https://mlsanity.com/paper/2402.19302/) (2024)
- [Deepzzle: Solving Visual Jigsaw Puzzles with Deep Learning andShortest Path Optimization](https://mlsanity.com/paper/2005.12548/) (2020)
- [Re-assembling the past: The RePAIR dataset and benchmark for real world 2D and 3D puzzle solving](https://mlsanity.com/paper/2410.24010/) (2024)
- [JigsawGAN: Auxiliary Learning for Solving Jigsaw Puzzles with Generative Adversarial Networks](https://mlsanity.com/paper/2101.07555/) (2021)
- [Solving Archaeological Puzzles](https://mlsanity.com/paper/1812.10553/) (2018)
- [A Genetic Algorithm-Based Solver for Very Large Jigsaw Puzzles](https://mlsanity.com/paper/1711.06769/) (2017)
- [Analyzing The Language of Visual Tokens](https://mlsanity.com/paper/2411.05001/) (2024)
- [Solving Masked Jigsaw Puzzles with Diffusion Vision Transformers](https://mlsanity.com/paper/2404.07292/) (2024)
- [From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval](https://mlsanity.com/paper/2502.12448/) (2025)
- [Pictorial and apictorial polygonal jigsaw puzzles from arbitrary number of crossing cuts](https://mlsanity.com/paper/2008.07644/) (2020)
- [GANzzle: Reframing jigsaw puzzle solving as a retrieval task using a generative mental image](https://mlsanity.com/paper/2207.05634/) (2022)
- [Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models](https://mlsanity.com/paper/2505.20728/) (2025)

## Built on by (most cited first, 4 of 4)

- [Do multimodal models imagine electric sheep?](https://mlsanity.com/paper/2605.09693/) (2026)
- [JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles](https://mlsanity.com/paper/2607.27670/) (2026)
- [Learning Permutation from Structure Without Supervision](https://mlsanity.com/paper/2605.25551/) (2026)
- [JigSync: Gauge-Resolved Synchronization for Jigsaw Reassembly under Unknown Piece Orientation](https://mlsanity.com/paper/2609.18379/) (2026)

## Abstract (arXiv)

Square jigsaw puzzles are typically solved by visually matching piece images to recover the original layout. This work introduces PuzLM, an alternative perspective that recasts jigsaw reassembly as a discrete sequence-to-sequence (Seq2Seq) problem, inspired by natural language representations. We design an efficient puzzle quantization procedure that transforms each piece into a short sequence of discrete tokens, enabling the direct application of standard Seq2Seq language models as powerful jigsaw solvers. Our approach demonstrates that accurate puzzle reconstruction can be achieved through purely symbolic reasoning over discrete representations, improving state-of-the-art performance even on puzzles with eroded boundaries or missing pieces.