# EZSR: Event-based Zero-Shot Recognition arXiv: https://arxiv.org/abs/2407.21616 ยท HTML: https://mlsanity.com/paper/2407.21616/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 36 of its 54 references that are in the map. - First posted on arXiv 31 Jul 2024. ## Builds on (most cited first, 30 of 36) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023) - [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018) - [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021) - [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023) - [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018) - [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021) - [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021) - [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021) - [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018) - [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023) - [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022) - [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020) - [Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades](https://mlsanity.com/paper/1507.07629/) (2015) - [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018) - [Events-to-Video: Bringing Modern Computer Vision to Event Cameras](https://mlsanity.com/paper/1904.08298/) (2019) - [N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras](https://mlsanity.com/paper/2112.01041/) (2021) - [Graph-Based Object Classification for Neuromorphic Vision Sensing](https://mlsanity.com/paper/1908.06648/) (2019) - [Event Camera Data Pre-training](https://mlsanity.com/paper/2301.01928/) (2023) - [HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors](https://mlsanity.com/paper/2211.09648/) (2022) - [Masked Event Modeling: Self-Supervised Pretraining for Event Cameras](https://mlsanity.com/paper/2212.10368/) (2022) - [Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies](https://mlsanity.com/paper/2304.12760/) (2023) - [ConvNet Architecture Search for Spatiotemporal Feature Learning](https://mlsanity.com/paper/1708.05038/) (2017) - [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024) - [EventCLIP: Adapting CLIP for Event-based Object Recognition](https://mlsanity.com/paper/2306.06354/) (2023) - [EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understanding](https://mlsanity.com/paper/2308.03135/) (2023) - [TMA: Temporal Motion Aggregation for Event-based Optical Flow](https://mlsanity.com/paper/2303.11629/) (2023) - [Label-Free Event-based Object Recognition via Joint Learning with Image Reconstruction from Events](https://mlsanity.com/paper/2308.09383/) (2023) ## Built on by (most cited first, 8 of 8) - [EventVL: Understand Event Streams via Multimodal Large Language Model](https://mlsanity.com/paper/2501.13707/) (2025) - [Scaling Dense Event-Stream Pretraining from Visual Foundation Models](https://mlsanity.com/paper/2603.03969/) (2026) - [DarkShake-DVS: Event-based Human Action Recognition under Low-light andShaking Camera Conditions](https://mlsanity.com/paper/2605.20680/) (2026) - [EventBench: Towards Comprehensive Benchmarking of Event-based MLLMs](https://mlsanity.com/paper/2511.18448/) (2025) - [RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding](https://mlsanity.com/paper/2605.19329/) (2026) - [Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras](https://mlsanity.com/paper/2609.06928/) (2026) - [Hyper-RED: Scalable Event Pre-training via Semantic Hypergraph Distillation](https://mlsanity.com/paper/2609.16811/) (2026) - [The Shape of Events: Edge-Based Inductive Biases via Cross-Domain Distillation](https://mlsanity.com/paper/2609.30478/) (2026) ## Abstract (arXiv) This paper studies zero-shot object recognition using event camera data. Guided by CLIP, which is pre-trained on RGB images, existing approaches achieve zero-shot object recognition by optimizing embedding similarities between event data and RGB images respectively encoded by an event encoder and the CLIP image encoder. Alternatively, several methods learn RGB frame reconstructions from event data for the CLIP image encoder. However, they often result in suboptimal zero-shot performance. This study develops an event encoder without relying on additional reconstruction networks. We theoretically analyze the performance bottlenecks of previous approaches: the embedding optimization objectives are prone to suffer from the spatial sparsity of event data, causing semantic misalignments between the learned event embedding space and the CLIP text embedding space. To mitigate the issue, we explore a scalar-wise modulation strategy. Furthermore, to scale up the number of events and RGB data pairs for training, we also study a pipeline for synthesizing event data from static RGB images in mass. Experimentally, we demonstrate an attractive scaling property in the number of parameters and synthesized data. We achieve superior zero-shot object recognition performance on extensive standard benchmark datasets, even compared with past supervised learning approaches. For example, our model with a ViT/B-16 backbone achieves 47.84% zero-shot accuracy on the N-ImageNet dataset.