# VELoRA: A Low-Rank Adaptation Approach for Efficient RGB-Event based Recognition

arXiv: https://arxiv.org/abs/2412.20064 · HTML: https://mlsanity.com/paper/2412.20064/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 61 references that are in the map.
- First posted on arXiv 28 Dec 2024.

## Builds on (most cited first, 19 of 19)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.10512/) (2023)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://mlsanity.com/paper/2210.07558/) (2022)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning](https://mlsanity.com/paper/2308.03303/) (2023)
- [DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding](https://mlsanity.com/paper/2411.14347/) (2024)
- [MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2402.12851/) (2024)
- [Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models](https://mlsanity.com/paper/2403.03432/) (2024)
- [ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models](https://mlsanity.com/paper/2403.16187/) (2024)
- [SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition](https://mlsanity.com/paper/2308.04369/) (2023)
- [Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2408.09764/) (2024)
- [SpikMamba: When SNN meets Mamba in Event-based Human Action Recognition](https://mlsanity.com/paper/2410.16746/) (2024)
- [Semantic-Aware Frame-Event Fusion based Pattern Recognition via Large Vision-Language Models](https://mlsanity.com/paper/2311.18592/) (2023)
- [Unleashing the Power of CNN and Transformer for Balanced RGB-Event Video Recognition](https://mlsanity.com/paper/2312.11128/) (2023)
- [Uncertainty-aware Bridge based Mobile-Former Network for Event-based Pattern Recognition](https://mlsanity.com/paper/2401.11123/) (2024)

## Built on by (most cited first, 2 of 2)

- [Dynamic Graph Induced Contour-aware Heat Conduction Network for Event-based Object Detection](https://mlsanity.com/paper/2505.12908/) (2025)
- [EventFace: Event-Based Face Recognition via Structure-Driven Spatiotemporal Modeling](https://mlsanity.com/paper/2604.06782/) (2026)

## Abstract (arXiv)

Pattern recognition leveraging both RGB and Event cameras can significantly enhance performance by deploying deep neural networks that utilize a fine-tuning strategy. Inspired by the successful application of large models, the introduction of such large models can also be considered to further enhance the performance of multi-modal tasks. However, fully fine-tuning these models leads to inefficiency and lightweight fine-tuning methods such as LoRA and Adapter have been proposed to achieve a better balance between efficiency and performance. To our knowledge, there is currently no work that has conducted parameter-efficient fine-tuning (PEFT) for RGB-Event recognition based on pre-trained foundation models. To address this issue, this paper proposes a novel PEFT strategy to adapt the pre-trained foundation vision models for the RGB-Event-based classification. Specifically, given the RGB frames and event streams, we extract the RGB and event features based on the vision foundation model ViT with a modality-specific LoRA tuning strategy. The frame difference of the dual modalities is also considered to capture the motion cues via the frame difference backbone network. These features are concatenated and fed into high-level Transformer layers for efficient multi-modal feature learning via modality-shared LoRA tuning. Finally, we concatenate these features and feed them into a classification head to achieve efficient fine-tuning. The source code and pre-trained models will be released on urlhttps://github.com/Event-AHU/VELoRA.