# VORTEX: Challenging CNNs at Texture Recognition by using Vision Transformers with Orderless and Randomized Token Encodings

arXiv: https://arxiv.org/abs/2503.06368 · HTML: https://mlsanity.com/paper/2503.06368/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 9 Mar 2025.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers](https://mlsanity.com/paper/2208.06366/) (2022)
- [From BoW to CNN: Two Decades of Texture Representation for Texture Classification](https://mlsanity.com/paper/1801.10324/) (2018)
- [ViTamin: Designing Scalable Vision Models in the Vision-Language Era](https://mlsanity.com/paper/2404.02132/) (2024)
- [Deep Texture Manifold for Ground Terrain Recognition](https://mlsanity.com/paper/1803.10896/) (2018)
- [Enriched CNN-Transformer Feature Aggregation Networks for Super-Resolution](https://mlsanity.com/paper/2203.07682/) (2022)
- [Bamboo: Building Mega-Scale Vision Dataset Continually with Human-Machine Synergy](https://mlsanity.com/paper/2203.07845/) (2022)
- [Multi-label Cluster Discrimination for Visual Representation Learning](https://mlsanity.com/paper/2407.17331/) (2024)
- [RADAM: Texture Recognition through Randomized Aggregated Encoding of Deep Activation Maps](https://mlsanity.com/paper/2303.04554/) (2023)
- [A Continual Development Methodology for Large-scale Multitask Dynamic ML Systems](https://mlsanity.com/paper/2209.07326/) (2022)
- [Fusion of complex networks and randomized neural networks for texture analysis](https://mlsanity.com/paper/1806.09170/) (2018)
- [Differential Viewpoints for Ground Terrain Material Recognition](https://mlsanity.com/paper/2009.11072/) (2020)
- [A Comparative Survey of Vision Transformers for Feature Extraction in Texture Analysis](https://mlsanity.com/paper/2406.06136/) (2024)
- [Spatio-spectral networks for color-texture analysis](https://mlsanity.com/paper/1909.06446/) (2019)
- [Multilayer deep feature extraction for visual texture recognition](https://mlsanity.com/paper/2208.10044/) (2022)

## Built on by (most cited first, 1 of 1)

- [HiPerViT: A Hierarchical Perceiver-Vision Transformer Architecture for Multi-Scale Texture Recognition](https://mlsanity.com/paper/2609.10917/) (2026)

## Abstract (arXiv)

Texture recognition has recently been dominated by ImageNet-pre-trained deep Convolutional Neural Networks (CNNs), with specialized modifications and feature engineering required to achieve state-of-the-art (SOTA) performance. However, although Vision Transformers (ViTs) were introduced a few years ago, little is known about their texture recognition ability. Therefore, in this work, we introduce VORTEX (ViTs with Orderless and Randomized Token Encodings for Texture Recognition), a novel method that enables the effective use of ViTs for texture analysis. VORTEX extracts multi-depth token embeddings from pre-trained ViT backbones and employs a lightweight module to aggregate hierarchical features and perform orderless encoding, obtaining a better image representation for texture recognition tasks. This approach allows seamless integration with any ViT with the common transformer architecture. Moreover, no fine-tuning of the backbone is performed, since they are used only as frozen feature extractors, and the features are fed to a linear SVM. We evaluate VORTEX on nine diverse texture datasets, demonstrating its ability to achieve or surpass SOTA performance in a variety of texture analysis scenarios. By bridging the gap between texture recognition with CNNs and transformer-based architectures, VORTEX paves the way for adopting emerging transformer foundation models. Furthermore, VORTEX demonstrates robust computational efficiency when coupled with ViT backbones compared to CNNs with similar costs. The method implementation and experimental scripts are publicly available in our online repository.