# Beyond Low-Rank: Low-Rank Sparse Prompting via Spiking Neural Network and Prompt Factorization

arXiv: https://arxiv.org/abs/2606.01945 · HTML: https://mlsanity.com/paper/2606.01945/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 62 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/1903.06530/) (2019)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Understanding and Improving Visual Prompting: A Label-Mapping Perspective](https://mlsanity.com/paper/2211.11635/) (2022)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)
- [Top-KAST: Top-K Always Sparse Training](https://mlsanity.com/paper/2106.03517/) (2021)
- [BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning](https://mlsanity.com/paper/2303.14773/) (2023)
- [E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning](https://mlsanity.com/paper/2307.13770/) (2023)
- [A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models](https://mlsanity.com/paper/2312.12730/) (2023)
- [Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-performance and Energy-efficient Object Detection](https://mlsanity.com/paper/2407.20708/) (2024)
- [Unleashing the Power of Visual Prompting At the Pixel Level](https://mlsanity.com/paper/2212.10556/) (2022)
- [Spiking PointNet: Spiking Neural Networks for Point Clouds](https://mlsanity.com/paper/2310.06232/) (2023)
- [AutoVP: An Automated Visual Prompting Framework and Benchmark](https://mlsanity.com/paper/2310.08381/) (2023)
- [Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction](https://mlsanity.com/paper/2303.09792/) (2023)
- [Direct Training High-Performance Deep Spiking Neural Networks: A Review of Theories and Methods](https://mlsanity.com/paper/2405.04289/) (2024)
- [OT-VP: Optimal Transport-guided Visual Prompting for Test-Time Adaptation](https://mlsanity.com/paper/2407.09498/) (2024)

## Abstract (arXiv)

Visual Prompting (VP) has emerged as an efficient paradigm for adapting large-scale pre-trained vision models to downstream tasks by incorporating learnable prompts at the input level. However, existing VP methods typically employ dense pixel-level prompts, which often suffer from redundant perturbations, limited generalization and energy inefficiency. To overcome these limitations, we propose to integrate brain-inspired spiking learning into visual prompt learning tasks. As we know that spiking neuron can perform inexpensive information processing by transmitting the input data into discrete spike trains and return sparse outputs. Inspired by this, we propose Low-Rank visual Spike Prompting (LoRSP), a novel framework that learns dynamic low-rank sparse visual prompts naturally via a Spiking neuron learning mechanism. The core idea of LoRSP is to exploit the brain-inspired sparse firing mechanism of spiking neurons to generate pixel-level sparse prompt for each instance. To be specific, we first construct a series of prompt factors via low-rank factorization to capture distinct prompt subspaces. These prompt factors are then fed into an SNN architecture, which performs the integrate-and-fire process to emit spikes. As a result, our LoRSP generates a sparse visual prompt while maintaining the low-rank constraint. This design enables instance-specific selective prompting, leading to more compact and robust adaptation across diverse downstream tasks. Extensive experiments on five heterogeneous vision backbones and multiple benchmarks demonstrate that LoRSP achieves competitive performance while requiring fewer tunable parameters compared to existing VP methods.