# Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles

arXiv: https://arxiv.org/abs/2306.00989 · HTML: https://mlsanity.com/paper/2306.00989/

## Key facts

- Cited by 290 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 66 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)

## Built on by (most cited first, 30 of 290)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning](https://mlsanity.com/paper/2603.14482/) (2026)
- [MedSAM2: Segment Anything in 3D Medical Images and Videos](https://mlsanity.com/paper/2504.03600/) (2025)
- [Streaming Dense Video Captioning](https://mlsanity.com/paper/2404.01297/) (2024)
- [When Do We Not Need Larger Vision Models?](https://mlsanity.com/paper/2403.13043/) (2024)
- [A Distractor-Aware Memory for Visual Object Tracking with SAM2](https://mlsanity.com/paper/2411.17576/) (2024)
- [SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentation](https://mlsanity.com/paper/2408.08870/) (2024)
- [Memory Consolidation Enables Long-Context Video Understanding](https://mlsanity.com/paper/2402.05861/) (2024)
- [DINO-Foresight: Looking into the Future with DINO](https://mlsanity.com/paper/2412.11673/) (2024)
- [Towards Privacy-Aware Sign Language Translation at Scale](https://mlsanity.com/paper/2402.09611/) (2024)
- [SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation](https://mlsanity.com/paper/2411.17646/) (2024)
- [Don't Look Twice: Faster Video Transformers with Run-Length Tokenization](https://mlsanity.com/paper/2411.05222/) (2024)
- [Segment Anything in Medical Images and Videos: Benchmark and Deployment](https://mlsanity.com/paper/2408.03322/) (2024)
- [Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects](https://mlsanity.com/paper/2403.16428/) (2024)
- [CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2411.10086/) (2024)
- [Prithvi WxC: Foundation Model for Weather and Climate](https://mlsanity.com/paper/2409.13598/) (2024)
- [OmniSAM: Omnidirectional Segment Anything Model for UDA in Panoramic Semantic Segmentation](https://mlsanity.com/paper/2503.07098/) (2025)
- [A Simple Baseline for Efficient Hand Mesh Reconstruction](https://mlsanity.com/paper/2403.01813/) (2024)
- [Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals](https://mlsanity.com/paper/2309.05927/) (2023)
- [Seeing What Matters: Generalizable AI-generated Video Detection with Forensic-Oriented Augmentation](https://mlsanity.com/paper/2506.16802/) (2025)
- [MULDE: Multiscale Log-Density Estimation via Denoising Score Matching for Video Anomaly Detection](https://mlsanity.com/paper/2403.14497/) (2024)
- [Efficient Track Anything](https://mlsanity.com/paper/2411.18933/) (2024)
- [An Empirical Study of Autoregressive Pre-training from Videos](https://mlsanity.com/paper/2501.05453/) (2025)
- [EdgeTAM: On-Device Track Anything Model](https://mlsanity.com/paper/2501.07256/) (2025)
- [VideoMAC: Video Masked Autoencoders Meet ConvNets](https://mlsanity.com/paper/2402.19082/) (2024)
- [Retrieval-augmented Few-shot Medical Image Segmentation with Foundation Models](https://mlsanity.com/paper/2408.08813/) (2024)
- [SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale](https://mlsanity.com/paper/2406.06907/) (2024)
- [CHAMMI: A benchmark for channel-adaptive models in microscopy imaging](https://mlsanity.com/paper/2310.19224/) (2023)
- [Window Attention is Bugged: How not to Interpolate Position Embeddings](https://mlsanity.com/paper/2311.05613/) (2023)

## Abstract (arXiv)

Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance. While these components lead to effective accuracies and attractive FLOP counts, the added complexity actually makes these transformers slower than their vanilla ViT counterparts. In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.