mlsanityData to 14 Oct 2026

TerViT: An Efficient Ternary Vision Transformer

arXiv 2201.08050 ↗First posted 20 Jan 2022cs.CV cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Vision transformers (ViTs) have demonstrated great potential in various visual tasks, but suffer from expensive computational and memory cost problems when deployed on resource-constrained devices. In this paper, we introduce a ternary vision transformer (TerViT) to ternarize the weights in ViTs, which are challenged by the large loss surface gap between real-valued and ternary parameters. To address the issue, we introduce a progressive training scheme by first training 8-bit transformers and then TerViT, and achieve a better optimization than conventional methods. Furthermore, we introduce channel-wise ternarization, by partitioning each matrix to different channels, each of which is with an unique distribution and ternarization interval. We apply our methods to popular DeiT and Swin backbones, and extensive results show that we can achieve competitive performance. For example, TerViT can quantize Swin-S to 13.1MB model size while achieving above 79% Top-1 accuracy on ImageNet dataset.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

22 of 27 references.

References · 22

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  7. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  9. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  10. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  11. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 315
  12. 2021Post-Training Quantization for Vision Transformercited by 278
10 more
  1. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  2. 2013Predicting Parameters in Deep Learningcited by 224
  3. 2016Ternary Weight Networkscited by 204
  4. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 193
  5. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  6. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  7. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  8. 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 60
  9. 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 52
  10. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 4; 2025: 3; 2026: 1.

11 papers.

  1. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  2. 2023Jumping through Local Minima: Quantization in the Loss Landscape of Vision Transformerscited by 18
  3. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Surveycited by 13
  4. 2024MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformercited by 10
  5. 2023TSPTQ-ViT: Two-scaled post-training quantization for vision transformercited by 7
  6. 2024Understanding Neural Network Binarization with Forward and Backward Proximal Quantizerscited by 4
  7. 2022CPT-V: A Contrastive Approach to Post-Training Quantization of Vision Transformerscited by 1
  8. 2025Efficient SAR Vessel Detection for FPGA-Based On-Satellite Sensingcited by 0
  9. 2025Vanishing Contributions: A Unified Framework for Smooth and Iterative Model Compressioncited by 0
  10. 2025Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compressioncited by 0
  11. 2026FTerViT: Fully Ternary Vision Transformercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ViT-1.58b: Mobile Vision Transformers in the 1-bit Era
  2. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
  3. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking
  4. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer
  5. 2022Unified Visual Transformer Compression
  6. 2024P2-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformer
  7. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformer
  8. 2024Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer