# SineLoRADelta: Sine-Activated Delta Compression

arXiv: https://arxiv.org/abs/2505.21895 · HTML: https://mlsanity.com/paper/2505.21895/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 62 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [KronA: Parameter Efficient Tuning with Kronecker Adapter](https://mlsanity.com/paper/2212.10650/) (2022)
- [Learned Point Cloud Geometry Compression](https://mlsanity.com/paper/1909.12037/) (2019)
- [Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy](https://mlsanity.com/paper/2307.16867/) (2023)
- [BitDelta: Your Fine-Tune May Only Be Worth One Bit](https://mlsanity.com/paper/2402.10193/) (2024)
- [Knowledge Composition using Task Vectors with Learned Anisotropic Scaling](https://mlsanity.com/paper/2407.02880/) (2024)
- [NOLA: Compressing LoRA using Linear Combination of Random Basis](https://mlsanity.com/paper/2310.02556/) (2023)
- [Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead](https://mlsanity.com/paper/2407.00066/) (2024)
- [Efficient Learning With Sine-Activated Low-rank Matrices](https://mlsanity.com/paper/2403.19243/) (2024)
- [DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs](https://mlsanity.com/paper/2312.05215/) (2023)
- [Fast Exact k-Means, k-Medians and Bregman Divergence Clustering in 1D](https://mlsanity.com/paper/1701.07204/) (2017)

## Abstract (arXiv)

Resource-constrained weight deployment is a task of immense practical importance. Recently, there has been interest in the specific task of textitDelta Compression, where parties each hold a common base model and only communicate compressed weight updates. However, popular parameter efficient updates such as Low Rank Adaptation (LoRA) face inherent representation limitations - which are especially pronounced when combined with aggressive quantization. To overcome this, we build on recent work that improves LoRA representation capacity by using fixed-frequency sinusoidal functions to increase stable rank without adding additional parameters. We extend this to the quantized setting and present the first theoretical analysis showing how stable rank evolves under quantization. From this, we introduce SineLoRADelta, a principled and effective method for delta compression that improves the expressivity of quantized low-rank adapters by applying a sinusoidal activation. We validate SineLoRADelta across a diverse variety of domains - including language modeling, vision-language tasks, and text-to-image generation - achieving up to 66% memory reduction with similar performance. We additionally provide a novel application of the canonical Bjontegaard Delta metric to consistently compare adapter compression changes across the rate-distortion curve.