# VLHSA: Vision-Language Hierarchical Semantic Alignment for Jigsaw Puzzle Solving with Eroded Gaps

arXiv: https://arxiv.org/abs/2509.25202 · HTML: https://mlsanity.com/paper/2509.25202/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Aligning Modalities in Vision Large Language Models via Preference Fine-tuning](https://mlsanity.com/paper/2402.11411/) (2024)
- [PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining](https://mlsanity.com/paper/2204.14095/) (2022)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement](https://mlsanity.com/paper/2405.15973/) (2024)
- [Deepzzle: Solving Visual Jigsaw Puzzles with Deep Learning andShortest Path Optimization](https://mlsanity.com/paper/2005.12548/) (2020)
- [Solving Archaeological Puzzles](https://mlsanity.com/paper/1812.10553/) (2018)
- [Solving Jigsaw Puzzles with Eroded Boundaries](https://mlsanity.com/paper/1912.00755/) (2019)
- [Solving Masked Jigsaw Puzzles with Diffusion Vision Transformers](https://mlsanity.com/paper/2404.07292/) (2024)
- [Logic.py: Bridging the Gap between LLMs and Constraint Solvers](https://mlsanity.com/paper/2502.15776/) (2025)
- [Solving Jigsaw Puzzles By the Graph Connection Laplacian](https://mlsanity.com/paper/1811.03188/) (2018)
- [Image Reassembly Combining Deep Learning and Shortest Path Problem](https://mlsanity.com/paper/1809.00898/) (2018)
- [ERL-MPP: Evolutionary Reinforcement Learning with Multi-head Puzzle Perception for Solving Large-scale Jigsaw Puzzles of Eroded Gaps](https://mlsanity.com/paper/2504.09608/) (2025)

## Built on by (most cited first, 4 of 4)

- [PuzLM: Solving Jigsaw Puzzles with Sequence-to-Sequence Language Models](https://mlsanity.com/paper/2511.06315/) (2025)
- [The Missing GAP: From Solving Square Jigsaw Puzzles to Handling Real World Archaeological Fragments](https://mlsanity.com/paper/2605.12077/) (2026)
- [JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles](https://mlsanity.com/paper/2607.27670/) (2026)
- [Fix the Match, Not the Solver: STITCH -- a Self-Supervised Compatibility Measure for Eroded Visual Puzzles](https://mlsanity.com/paper/2610.13176/) (2026)

## Abstract (arXiv)

Jigsaw puzzle solving remains challenging in computer vision, requiring an understanding of both local fragment details and global spatial relationships. While most traditional approaches only focus on visual cues like edge matching and visual coherence, few methods explore natural language descriptions for semantic guidance in challenging scenarios, especially for eroded gap puzzles. We propose a vision-language framework that leverages textual context to enhance puzzle assembly performance. Our approach centers on the Vision-Language Hierarchical Semantic Alignment (VLHSA) module, which aligns visual patches with textual descriptions through multi-level semantic matching from local tokens to global context. Also, a multimodal architecture that combines dual visual encoders with language features for cross-modal reasoning is integrated into this module. Experiments demonstrate that our method significantly outperforms state-of-the-art models across various datasets, achieving substantial improvements, including a 14.2 percentage point gain in piece accuracy. Ablation studies confirm the critical role of the VLHSA module in driving improvements over vision-only approaches. Our work establishes a new paradigm for jigsaw puzzle solving by incorporating multimodal semantic insights.