mlsanityData to 14 Oct 2026

ViT-1.58b: Mobile Vision Transformers in the 1-bit Era

arXiv 2406.18051 ↗First posted 26 Jun 2024cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViTs) have achieved remarkable performance in various image classification tasks by leveraging the attention mechanism to process image patches as tokens. However, the high computational and memory demands of ViTs pose significant challenges for deployment in resource-constrained environments. This paper introduces ViT-1.58b, a novel 1.58-bit quantized ViT model designed to drastically reduce memory and computational overhead while preserving competitive performance. ViT-1.58b employs ternary quantization, which refines the balance between efficiency and accuracy by constraining weights to {-1, 0, 1} and quantizing activations to 8-bit precision. Our approach ensures efficient scaling in terms of both memory and computation. Experiments on CIFAR-10 and ImageNet-1k demonstrate that ViT-1.58b maintains comparable accuracy to full-precision Vit, with significant reductions in memory usage and computational costs. This paper highlights the potential of extreme quantization techniques in developing sustainable AI solutions and contributes to the broader discourse on efficient model deployment in practical applications. Our code and weights are available at https://github.com/DLYuanGod/ViT-1.58b.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

19 of 25 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2016Layer Normalizationcited by 5,669
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  8. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  9. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  10. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  11. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 151
  12. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
7 more
  1. 2021Learning Efficient Vision Transformers via Fine-Grained Manifold Distillationcited by 60
  2. 2018Value-aware Quantization for Training and Inference of Neural Networkscited by 52
  3. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  4. 2022CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Predictioncited by 39
  5. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
  6. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
  7. 2022VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformercited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 1.

8 papers.

  1. 2025TerEffic: Highly Efficient Ternary LLM Inference on FPGAcited by 6
  2. 2024Mixed Non-linear Quantization for Vision Transformerscited by 3
  3. 2024An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networkscited by 3
  4. 2025Membership Inference Risks in Quantized Models: A Theoretical and Empirical Studycited by 3
  5. 2024Efficient Ternary Weight Embedding Model: Bridging Scalability and Performancecited by 1
  6. 2025Auto-US: An Ultrasound Video Diagnosis Agent Using Video Classification Framework and LLMscited by 1
  7. 2025R2Q: Towards Robust 2-Bit Large Language Models via Residual Refinement Quantizationcited by 0
  8. 2026FTerViT: Fully Ternary Vision Transformercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
  2. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
  3. 2024Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer
  4. 2024P2-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformer
  5. 2023Bi-ViT: Pushing the Limit of Vision Transformer Quantization
  6. 2024PQV-Mobile: A Combined Pruning and Quantization Toolkit to Optimize Vision Transformers for Mobile Applications
  7. 2023I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantization
  8. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformer