mlsanityData to 7 Oct 2026

VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks

arXiv 2403.00522 ↗First posted 1 Mar 2024cs.CVcited by 28

Not on the map. Why?

Abstract (arXiv)

Large language models are built on top of a transformer-based architecture to process textual inputs. For example, the LLaMA stands out among many open-source implementations. Can the same transformer be used to process 2D images? In this paper, we answer this question by unveiling a LLaMA-like vision transformer in plain and pyramid forms, termed VisionLLaMA, which is tailored for this purpose. VisionLLaMA is a unified and generic modelling framework for solving most vision tasks. We extensively evaluate its effectiveness using typical pre-training paradigms in a good portion of downstream tasks of image perception and especially image generation. In many cases, VisionLLaMA have exhibited substantial gains over the previous state-of-the-art vision transformers. We believe that VisionLLaMA can serve as a strong new baseline model for vision generation and understanding. Our code is released at https://github.com/Meituan-AutoML/VisionLLaMA.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 66 of its 79 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

66 of 79 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
38 more
  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2017Mask R-CNNcited by 7,259
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2016Layer Normalizationcited by 5,579
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  13. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  14. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  15. 2013Auto-Encoding Variational Bayescited by 3,980
  16. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  17. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  18. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  19. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  20. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  21. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  22. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  23. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  24. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  25. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  26. 2020GLU Variants Improve Transformercited by 1,724
  27. 2019Root Mean Square Layer Normalizationcited by 1,425
  28. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  29. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  30. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  31. 2018Self-Attention with Relative Position Representationscited by 1,112
  32. 2016Deep Networks with Stochastic Depthcited by 973
  33. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  34. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  35. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  36. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  37. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  38. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 18; 2026: 6.

28 papers.

Sort:

By area:

  1. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  2. 2025FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editingcited by 44
  3. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 42
  4. 2025Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Modelscited by 35
  5. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  6. 2024SketchAgent: Language-Driven Sequential Sketch Generationcited by 30
  7. 2025Omni-Effects: Unified and Spatially-Controllable Visual Effects Generationcited by 27
  8. 2025USP: Unified Self-Supervised Pretraining for Image Generation and Understandingcited by 24
  9. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 23
  10. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 21
  11. 2025From Editor to Dense Geometry Estimatorcited by 20
  12. 2025X2I: Seamless Integration of Multimodal Understanding into Diffusion Transformer via Attention Distillationcited by 11
16 more
  1. 2025Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?cited by 9
  2. 2025ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraintscited by 9
  3. 2026ViT-5: Vision Transformers for The Mid-2020scited by 7
  4. 2025Eevee: Towards Close-up High-resolution Video-based Virtual Try-oncited by 5
  5. 2025Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encodingcited by 4
  6. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
  7. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Modelscited by 1
  8. 2026Unveiling the Secret of AdaLN-Zero in Diffusion Transformercited by 1
  9. 2025HalCECE: A Framework for Explainable Hallucination Detection through Conceptual Counterfactuals in Image Captioningcited by 0
  10. 2025CA^2ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognitioncited by 0
  11. 2025LOOPE: Learnable Optimal Patch Order in Positional Embeddings for Vision Transformerscited by 0
  12. 2025PiT: Progressive Diffusion Transformercited by 0
  13. 2026Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, Generalcited by 0
  14. 2026BLUEPRINT Rebuilding a Legacy: Multimodal Retrieval for Complex Engineering Drawings and Documentscited by 0
  15. 2026Weierstrass Positional Encoding for Vision Transformerscited by 0
  16. 2026MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perceptioncited by 0