mlsanityData to 14 Oct 2026

Deeper Inside Deep ViT

arXiv 2508.04181 ↗First posted 6 Aug 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

There have been attempts to create large-scale structures in vision models similar to LLM, such as ViT-22B. While this research has provided numerous analyses and insights, our understanding of its practical utility remains incomplete. Therefore, we examine how this model structure reacts and train in a local environment. We also highlight the instability in training and make some model modifications to stabilize it. The ViT-22B model, trained from scratch, overall outperformed ViT in terms of performance under the same parameter size. Additionally, we venture into the task of image generation, which has not been attempted in ViT-22B. We propose an image generation architecture using ViT and investigate which between ViT and ViT-22B is a more suitable structure for image generation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 31 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

27 of 31 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2016Layer Normalizationcited by 5,669
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
15 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  2. 2017Mixed Precision Trainingcited by 1,156
  3. 2021Scaling Vision Transformerscited by 999
  4. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  5. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  6. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  7. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  8. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 607
  9. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  10. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  11. 2018Generative Adversarial Network in Medical Imaging: A Reviewcited by 257
  12. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  13. 2021BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generationcited by 47
  14. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 41
  15. 2021Generative Adversarial U-Net for Domain-free Medical Image Augmentationcited by 3

Built on by

Not cited yet.