mlsanityData to 14 Oct 2026

Mixed-precision Supernet Training from Vision Foundation Models using Low Rank Adapter

arXiv 2403.20080 ↗First posted 29 Mar 2024cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Compression of large and performant vision foundation models (VFMs) into arbitrary bit-wise operations (BitOPs) allows their deployment on various hardware. We propose to fine-tune a VFM to a mixed-precision quantized supernet. The supernet-based neural architecture search (NAS) can be adopted for this purpose, which trains a supernet, and then subnets within arbitrary hardware budgets can be extracted. However, existing methods face difficulties in optimizing the mixed-precision search space and incurring large memory costs during training. To tackle these challenges, first, we study the effective search space design for fine-tuning a VFM by comparing different operators (such as resolution, feature size, width, depth, and bit-widths) in terms of performance and BitOPs reduction. Second, we propose memory-efficient supernet training using a low-rank adapter (LoRA) and a progressive training strategy. The proposed method is evaluated for the recently proposed VFM, Segment Anything Model, fine-tuned on segmentation tasks. The searched model yields about a 95% reduction in BitOPs without incurring performance degradation.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

33 of 38 references.

References · 33

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Segment Anythingcited by 8,234
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  6. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  7. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  8. 2023Segment Everything Everywhere All at Oncecited by 548
  9. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  10. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  11. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  12. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
21 more
  1. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  2. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 188
  3. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  4. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  5. 2019FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Searchcited by 130
  6. 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 125
  7. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  8. 2020Bayesian Bits: Unifying Quantization and Pruningcited by 94
  9. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  10. 2021Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimizationcited by 61
  11. 2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 54
  12. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  13. 2021Searching the Search Space of Vision Transformercited by 41
  14. 2021HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformerscited by 39
  15. 2020Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Searchcited by 26
  16. 2021UniNet: Unified Architecture Search with Convolution, Transformer, and MLPcited by 23
  17. 2021BatchQuant: Quantized-for-all Architecture Search with Robust Quantizercited by 20
  18. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  19. 2023ElasticViT: Conflict-aware Supernet Training for Deploying Fast Vision Transformer on Diverse Mobile Devicescited by 10
  20. 2021Searching for Efficient Multi-Stage Vision Transformerscited by 9
  21. 2023ShiftNAS: Improving One-shot NAS via Probability Shiftcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026LLM as a Tool, Not an Agent: Code-Mined Tree Transformations for Neural Architecture Searchcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023Free Bits: Latency Optimization of Mixed-Precision Quantized Neural Networks on the Edge
  2. 2025Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models
  3. 2023SimQ-NAS: Simultaneous Quantization Policy and Neural Architecture Search
  4. 2021Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
  5. 2024Scaling Up Quantization-Aware Neural Architecture Search for Efficient Deep Learning on the Edge
  6. 2024Joint Pruning and Channel-wise Mixed-Precision Quantization for Efficient Deep Neural Networks
  7. 2022Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes
  8. 2025RAM-NAS: Resource-aware Multiobjective Neural Architecture Search Method for Robot Vision Tasks