# LLM-EvRep: Learning an LLM-Compatible Event Representation Using a Self-Supervised Framework

arXiv: https://arxiv.org/abs/2502.14273 · HTML: https://mlsanity.com/paper/2502.14273/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning](https://mlsanity.com/paper/2403.11401/) (2024)
- [Recurrent Vision Transformers for Object Detection with Event Cameras](https://mlsanity.com/paper/2212.05598/) (2022)
- [Asynchronous, Photometric Feature Tracking using Events and Frames](https://mlsanity.com/paper/1807.09713/) (2018)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras](https://mlsanity.com/paper/2112.01041/) (2021)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [Event-based, 6-DOF Camera Tracking from Photometric Depth Maps](https://mlsanity.com/paper/1607.03468/) (2016)
- [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024)
- [EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understanding](https://mlsanity.com/paper/2308.03135/) (2023)
- [EventPoint: Self-Supervised Interest Point Detection and Description for Event-based Camera](https://mlsanity.com/paper/2109.00210/) (2021)
- [Chain of Visual Perception: Harnessing Multimodal Large Language Models for Zero-shot Camouflaged Object Detection](https://mlsanity.com/paper/2311.11273/) (2023)
- [Are High-Resolution Event Cameras Really Needed?](https://mlsanity.com/paper/2203.14672/) (2022)
- [E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning](https://mlsanity.com/paper/2401.08117/) (2024)
- [Multi-modal Large Language Model Enhanced Pseudo 3D Perception Framework for Visual Commonsense Reasoning](https://mlsanity.com/paper/2301.13335/) (2023)
- [Can Large Language Models Grasp Event Signals? Exploring Pure Zero-Shot Event-based Recognition](https://mlsanity.com/paper/2409.09628/) (2024)

## Built on by (most cited first, 3 of 3)

- [A Survey of 3D Reconstruction with Event Cameras](https://mlsanity.com/paper/2505.08438/) (2025)
- [A Survey on Event-driven 3D Reconstruction: Development under Different Categories](https://mlsanity.com/paper/2503.19753/) (2025)
- [Revealing Latent Information: A Physics-inspired Self-supervised Pre-training Framework for Noisy and Sparse Events](https://mlsanity.com/paper/2508.05507/) (2025)

## Abstract (arXiv)

Recent advancements in event-based recognition have demonstrated significant promise, yet most existing approaches rely on extensive training, limiting their adaptability for efficient processing of event-driven visual content. Meanwhile, large language models (LLMs) have exhibited remarkable zero-shot capabilities across diverse domains, but their application to event-based visual recognition remains largely unexplored. To bridge this gap, we propose LLM-EvGen, an event representation generator that produces LLM-compatible event representations LLM-EvRep, thereby enhancing the performance of LLMs on event recognition tasks. The generator is trained using a self-supervised framework, aligning the generated representations with semantic consistency and structural fidelity. Comprehensive experiments were conducted on three datasets: N-ImageNet, N-Caltech101, and N-MNIST. The results demonstrate that our method, LLM-EvRep, outperforms the event-to-video method, E2VID, by 15.93%, 0.82%, and 50.21%, respectively, in recognition tasks when evaluated using GPT-4o.