# Re-assembling the past: The RePAIR dataset and benchmark for real world 2D and 3D puzzle solving

arXiv: https://arxiv.org/abs/2410.24010 · HTML: https://mlsanity.com/paper/2410.24010/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 98 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [TEASER: Fast and Certifiable Point Cloud Registration](https://mlsanity.com/paper/2001.07715/) (2020)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021)
- [PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes](https://mlsanity.com/paper/1911.10949/) (2019)
- [JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints](https://mlsanity.com/paper/2111.12772/) (2021)
- [IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks](https://mlsanity.com/paper/1911.07246/) (2019)
- [Generative 3D Part Assembly via Dynamic Graph Learning](https://mlsanity.com/paper/2006.07793/) (2020)
- [Learning 3D Part Assembly from a Single Image](https://mlsanity.com/paper/2003.09754/) (2020)
- [Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors](https://mlsanity.com/paper/2205.14886/) (2022)
- [Breaking Bad: A Dataset for Geometric Fracture and Reassembly](https://mlsanity.com/paper/2210.11463/) (2022)
- [RGL-NET: A Recurrent Graph Learning framework for Progressive Part Assembly](https://mlsanity.com/paper/2107.12859/) (2021)
- [DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly](https://mlsanity.com/paper/2402.19302/) (2024)
- [Jigsaw: Learning to Assemble Multiple Fractured Objects](https://mlsanity.com/paper/2305.17975/) (2023)
- [Leveraging SE(3) Equivariance for Learning 3D Geometric Shape Assembly](https://mlsanity.com/paper/2309.06810/) (2023)
- [Deepzzle: Solving Visual Jigsaw Puzzles with Deep Learning andShortest Path Optimization](https://mlsanity.com/paper/2005.12548/) (2020)
- [Fantastic Breaks: A Dataset of Paired 3D Scans of Real-World Broken Objects and Their Complete Counterparts](https://mlsanity.com/paper/2303.14152/) (2023)
- [Solving Archaeological Puzzles](https://mlsanity.com/paper/1812.10553/) (2018)
- [A Genetic Algorithm-Based Solver for Very Large Jigsaw Puzzles](https://mlsanity.com/paper/1711.06769/) (2017)
- [3D Reconstruction of Incomplete Archaeological Objects Using a Generative Adversarial Network](https://mlsanity.com/paper/1711.06363/) (2017)
- [Solving Jigsaw Puzzles with Linear Programming](https://mlsanity.com/paper/1511.04472/) (2015)
- [Pictorial and apictorial polygonal jigsaw puzzles from arbitrary number of crossing cuts](https://mlsanity.com/paper/2008.07644/) (2020)
- [GANzzle: Reframing jigsaw puzzle solving as a retrieval task using a generative mental image](https://mlsanity.com/paper/2207.05634/) (2022)
- [JigsawNet: Shredded Image Reassembly using Convolutional Neural Network and Loop-based Composition](https://mlsanity.com/paper/1809.04137/) (2018)
- [A Generalized Genetic Algorithm-Based Solver for Very Large Jigsaw Puzzles of Complex Types](https://mlsanity.com/paper/1711.06768/) (2017)
- [DeepMend: Learning Occupancy Functions to Represent Shape for Repair](https://mlsanity.com/paper/2210.05728/) (2022)
- [Assemble Them All: Physics-Based Planning for Generalizable Assembly by Disassembly](https://mlsanity.com/paper/2211.03977/) (2022)
- [A Novel Hybrid Scheme Using Genetic Algorithms and Deep Learning for the Reconstruction of Portuguese Tile Panels](https://mlsanity.com/paper/1912.02707/) (2019)
- [DeepJoin: Learning a Joint Occupancy, Signed Distance, and Normal Field Function for Shape Repair](https://mlsanity.com/paper/2211.12400/) (2022)

## Built on by (most cited first, 19 of 19)

- [GARF: Learning Generalizable 3D Reassembly for Real-World Fractures](https://mlsanity.com/paper/2504.05400/) (2025)
- [Pairwise Alignment & Compatibility for Arbitrarily Irregular Image Fragments](https://mlsanity.com/paper/2507.09767/) (2025)
- [VaseVQA: Multimodal Agent and Benchmark for Ancient Greek Pottery](https://mlsanity.com/paper/2509.17191/) (2025)
- [BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly](https://mlsanity.com/paper/2506.06221/) (2025)
- [PuzLM: Solving Jigsaw Puzzles with Sequence-to-Sequence Language Models](https://mlsanity.com/paper/2511.06315/) (2025)
- [The Missing GAP: From Solving Square Jigsaw Puzzles to Handling Real World Archaeological Fragments](https://mlsanity.com/paper/2605.12077/) (2026)
- [DM-OSVP++: One-Shot View Planning Using 3D Diffusion Models for Active RGB-Based Object Reconstruction](https://mlsanity.com/paper/2504.11674/) (2025)
- [ReassembleNet: Learnable Keypoints and Diffusion for 2D Fresco Reconstruction](https://mlsanity.com/paper/2505.21117/) (2025)
- [Learning Generalizable Shape Completion with SIM(3) Equivariance](https://mlsanity.com/paper/2509.26631/) (2025)
- [Rejoining fragmented ancient bamboo slips with physics-driven deep learning](https://mlsanity.com/paper/2505.08601/) (2025)
- [Interactive Shaping of Granular Media Using Reinforcement Learning](https://mlsanity.com/paper/2509.06469/) (2025)
- [Solving Convex Partition Visual Jigsaw Puzzles](https://mlsanity.com/paper/2511.04450/) (2025)
- [E-M3RF: An Equivariant Multimodal 3D Re-assembly Framework](https://mlsanity.com/paper/2511.21422/) (2025)
- [Solving Jigsaw Puzzles in the Wild: Human-Guided Reconstruction of Cultural Heritage Fragments](https://mlsanity.com/paper/2603.06389/) (2026)
- [Benchmarking Content-Based Puzzle Solvers on Corrupted Jigsaw Puzzles](https://mlsanity.com/paper/2507.07828/) (2025)
- [Unified Primitive Proxies for Structured Shape Completion](https://mlsanity.com/paper/2601.00759/) (2026)
- [JigSync: Gauge-Resolved Synchronization for Jigsaw Reassembly under Unknown Piece Orientation](https://mlsanity.com/paper/2609.18379/) (2026)
- [Fragment-Aware Vision Transformers for Fresco-Fragment Style Classification](https://mlsanity.com/paper/2609.21012/) (2026)
- [Fix the Match, Not the Solver: STITCH -- a Self-Supervised Compatibility Measure for Eroded Visual Puzzles](https://mlsanity.com/paper/2610.13176/) (2026)

## Abstract (arXiv)

This paper proposes the RePAIR dataset that represents a challenging benchmark to test modern computational and data driven methods for puzzle-solving and reassembly tasks. Our dataset has unique properties that are uncommon to current benchmarks for 2D and 3D puzzle solving. The fragments and fractures are realistic, caused by a collapse of a fresco during a World War II bombing at the Pompeii archaeological park. The fragments are also eroded and have missing pieces with irregular shapes and different dimensions, challenging further the reassembly algorithms. The dataset is multi-modal providing high resolution images with characteristic pictorial elements, detailed 3D scans of the fragments and meta-data annotated by the archaeologists. Ground truth has been generated through several years of unceasing fieldwork, including the excavation and cleaning of each fragment, followed by manual puzzle solving by archaeologists of a subset of approx. 1000 pieces among the 16000 available. After digitizing all the fragments in 3D, a benchmark was prepared to challenge current reassembly and puzzle-solving methods that often solve more simplistic synthetic scenarios. The tested baselines show that there clearly exists a gap to fill in solving this computationally complex problem.