mlsanityData to 14 Oct 2026

Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning

arXiv 2605.24058 ↗First posted 22 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

On-device adaptation of large language models commonly keeps a quantized base model frozen while training and deploying a small, task-specific LoRA adapter. In the unmerged adapter-mode setting, however, the adapter is more than a compact storage module; it introduces an additional dense floating-point branch, maintains a trainable state for local updates, and acts as a unit of communication and hot-swapping.We introduce LoRDBA, a LoRA-compatible adapter that replaces both low-rank factors with binary sign carriers while representing magnitudes through lightweight, channel-wise scales, converting the dense adapter branch into two sign-accumulation matrix multiplications interleaved with channel-wise scaling. A finite-sample analysis shows that reconstruction quality is governed by the residual-to-magnitude ratio of the original LoRA factors. In adapter-mode experiments, LoRDBA outperforms low-bit baselines at matched model sizes while matching fp16 LoRA quality in selected regimes. The unmerged adapter incurs at most 8% prefill latency overhead at matched rank r=16 despite an over 10x reduction in adapter footprint, with moderate training memory overhead of approximately 1.6x that of fp16 LoRA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

28 of 39 references.

References · 28

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  4. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  6. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  7. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  8. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  9. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  10. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  11. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  12. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
16 more
  1. 2015Global Convergence of ADMM in Nonconvex Nonsmooth Optimizationcited by 98
  2. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  3. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  4. 2017Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMMcited by 61
  5. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 41
  6. 2024BitDelta: Your Fine-Tune May Only Be Worth One Bitcited by 39
  7. 2017Douglas-Rachford splitting and ADMM for nonconvex optimization: tight convergence resultscited by 21
  8. 2023Controlling Continuous Relaxation for Combinatorial Optimizationcited by 16
  9. 2024Optimization by Parallel Quasi-Quantum Annealing with Gradient-Based Samplingcited by 7
  10. 2024Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorizationcited by 6
  11. 2025LPCD: Unified Framework from Layer-Wise to Submodule Quantizationcited by 6
  12. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimatorcited by 4
  13. 2025Addition is almost all you need: Compressing large language models with double binary factorizationcited by 3
  14. 2025More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantizationcited by 3
  15. 2026OneComp: One-Line Revolution for Generative AI Model Compressioncited by 3
  16. 2025LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bitscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
  2. 2023INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation
  3. 2024Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance
  4. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  5. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  6. 2025FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
  7. 2026Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation
  8. 2025LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits