# Hypergraph Vision Transformers: Images are More than Nodes, More than Edges

arXiv: https://arxiv.org/abs/2504.08710 · HTML: https://mlsanity.com/paper/2504.08710/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 66 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)

## Built on by (most cited first, 16 of 16)

- [YOLOv13: Real-Time Object Detection with Hypergraph-Enhanced Adaptive Visual Perception](https://mlsanity.com/paper/2506.17733/) (2025)
- [Commonality in Few: Few-Shot Multimodal Anomaly Detection via Hypergraph-Enhanced Memory](https://mlsanity.com/paper/2511.05966/) (2025)
- [DHG-Bench: A Comprehensive Benchmark for Deep Hypergraph Learning](https://mlsanity.com/paper/2508.12244/) (2025)
- [SoftHGNN: Soft Hypergraph Neural Networks for General Visual Recognition](https://mlsanity.com/paper/2505.15325/) (2025)
- [HAMSA: Scanning-Free Vision State Space Models via SpectralPulseNet](https://mlsanity.com/paper/2604.14724/) (2026)
- [HGTS-Former: Hierarchical HyperGraph Transformer for Multivariate Time Series Analysis](https://mlsanity.com/paper/2508.02411/) (2025)
- [AttentionViG: Cross-Attention-Based Dynamic Neighbor Aggregation in Vision GNNs](https://mlsanity.com/paper/2509.25570/) (2025)
- [Efficiency Follows Global-Local Decoupling](https://mlsanity.com/paper/2603.19567/) (2026)
- [Learning Label-Efficient Interpretable Medical Image Diagnosis via Semi-supervised Hypergraph Concept Bottleneck Model](https://mlsanity.com/paper/2606.01698/) (2026)
- [Hybrid Hypergraph Networks for Multimodal Sequence Data Classification](https://mlsanity.com/paper/2508.00926/) (2025)
- [HyperPotter: Spell the Charm of High-Order Interactions in Audio Deepfake Detection](https://mlsanity.com/paper/2602.05670/) (2026)
- [Hyper-FEOD: Sparse Hypergraph-Enhanced Frame-Event Object Detection with Fine-Grained MoE](https://mlsanity.com/paper/2604.11140/) (2026)
- [H2VLR: Heterogeneous Hypergraph Vision-Language Reasoning for Few-Shot Anomaly Detection](https://mlsanity.com/paper/2604.14507/) (2026)
- [Anchor-guided Hypergraph Condensation with Dual-level Discrimination](https://mlsanity.com/paper/2605.10001/) (2026)
- [Structured Hyperedge Adaptation for Parameter-Efficient Fine-Tuning of Vision Transformers](https://mlsanity.com/paper/2606.22383/) (2026)
- [Fragment-Aware Vision Transformers for Fresco-Fragment Style Classification](https://mlsanity.com/paper/2609.21012/) (2026)

## Abstract (arXiv)

Recent advancements in computer vision have highlighted the scalability of Vision Transformers (ViTs) across various tasks, yet challenges remain in balancing adaptability, computational efficiency, and the ability to model higher-order relationships. Vision Graph Neural Networks (ViGs) offer an alternative by leveraging graph-based methodologies but are hindered by the computational bottlenecks of clustering algorithms used for edge generation. To address these issues, we propose the Hypergraph Vision Transformer (HgVT), which incorporates a hierarchical bipartite hypergraph structure into the vision transformer framework to capture higher-order semantic relationships while maintaining computational efficiency. HgVT leverages population and diversity regularization for dynamic hypergraph construction without clustering, and expert edge pooling to enhance semantic extraction and facilitate graph-based image retrieval. Empirical results demonstrate that HgVT achieves strong performance on image classification and retrieval, positioning it as an efficient framework for semantic-based vision tasks.