# ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More

arXiv: https://arxiv.org/abs/2403.12534 · HTML: https://mlsanity.com/paper/2403.12534/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [A Survey of Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2202.10936/) (2022)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [TAM: Temporal Adaptive Module for Video Recognition](https://mlsanity.com/paper/2005.06803/) (2020)
- [Data Uncertainty Learning in Face Recognition](https://mlsanity.com/paper/2003.11339/) (2020)
- [Core Challenges of Social Robot Navigation: A Survey](https://mlsanity.com/paper/2103.05668/) (2021)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [Temporal Binary Representation for Event-Based Action Recognition](https://mlsanity.com/paper/2010.08946/) (2020)
- [E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition](https://mlsanity.com/paper/2112.03596/) (2021)
- [Event Transformer. A sparse-aware solution for efficient event data processing](https://mlsanity.com/paper/2204.03355/) (2022)
- [Speed Invariant Time Surface for Learning to Detect Corner Points with Event-Based Cameras](https://mlsanity.com/paper/1903.11332/) (2019)
- [View-Invariant Probabilistic Embedding for Human Pose](https://mlsanity.com/paper/1912.01001/) (2019)
- [HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors](https://mlsanity.com/paper/2211.09648/) (2022)
- [MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model](https://mlsanity.com/paper/2210.05335/) (2022)
- [A Voxel Graph CNN for Object Classification with Event Cameras](https://mlsanity.com/paper/2106.00216/) (2021)
- [EventCLIP: Adapting CLIP for Event-based Object Recognition](https://mlsanity.com/paper/2306.06354/) (2023)

## Built on by (most cited first, 30 of 33)

- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities](https://mlsanity.com/paper/2407.11351/) (2024)
- [EventGPT: Event Stream Understanding with Multimodal Large Language Models](https://mlsanity.com/paper/2412.00832/) (2024)
- [Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentation](https://mlsanity.com/paper/2407.11344/) (2024)
- [UCF-Crime-DVS: A Novel Event-Based Dataset for Video Anomaly Detection with Spiking Neural Networks](https://mlsanity.com/paper/2503.12905/) (2025)
- [EventVL: Understand Event Streams via Multimodal Large Language Model](https://mlsanity.com/paper/2501.13707/) (2025)
- [Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2408.09764/) (2024)
- [MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selection](https://mlsanity.com/paper/2412.16876/) (2024)
- [MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for Multi-modal Semantic Segmentation](https://mlsanity.com/paper/2503.06700/) (2025)
- [EventFly: Event Camera Perception from Ground to the Sky](https://mlsanity.com/paper/2503.19916/) (2025)
- [Reducing Unimodal Bias in Multi-Modal Semantic Segmentation with Multi-Scale Functional Entropy Regularization](https://mlsanity.com/paper/2505.06635/) (2025)
- [Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras](https://mlsanity.com/paper/2507.17664/) (2025)
- [CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework](https://mlsanity.com/paper/2504.12576/) (2025)
- [CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition](https://mlsanity.com/paper/2410.07153/) (2024)
- [EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More](https://mlsanity.com/paper/2407.01884/) (2024)
- [Entropy Heat-Mapping: Localizing GPT-Based OCR Errors with Sliding-Window Shannon Analysis](https://mlsanity.com/paper/2505.00746/) (2025)
- [DarkShake-DVS: Event-based Human Action Recognition under Low-light andShaking Camera Conditions](https://mlsanity.com/paper/2605.20680/) (2026)
- [EventDance++: Language-guided Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition](https://mlsanity.com/paper/2409.12778/) (2024)
- [PASS: Path-selective State Space Model for Event-based Recognition](https://mlsanity.com/paper/2409.16953/) (2024)
- [Helios 2.0: A Robust, Ultra-Low Power Gesture Recognition System Optimised for Event-Sensor based Wearables](https://mlsanity.com/paper/2503.07825/) (2025)
- [FLAMES: A Hybrid Spiking-State Space Model for Adaptive Memory Retention in Event-Based Learning](https://mlsanity.com/paper/2504.01257/) (2025)
- [EGFormer: Towards Efficient and Generalizable Multimodal Semantic Segmentation](https://mlsanity.com/paper/2505.14014/) (2025)
- [Visual Grounding from Event Cameras](https://mlsanity.com/paper/2509.09584/) (2025)
- [EventBench: Towards Comprehensive Benchmarking of Event-based MLLMs](https://mlsanity.com/paper/2511.18448/) (2025)
- [SMV-EAR: Bring Spatiotemporal Multi-View Representation Learning into Efficient Event-Based Action Recognition](https://mlsanity.com/paper/2601.17391/) (2026)
- [Ego-Human Motion Prediction with 3D-Aware LLM](https://mlsanity.com/paper/2607.07001/) (2026)
- [SARTM: Segment Any RGB Thermal Model with Language aided Distillation](https://mlsanity.com/paper/2505.01950/) (2025)
- [Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration](https://mlsanity.com/paper/2505.11895/) (2025)
- [BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2506.03675/) (2025)
- [PSTTS: A Plug-and-Play Token Selector for Efficient Event-based Spatio-temporal Representation Learning](https://mlsanity.com/paper/2509.22481/) (2025)

## Abstract (arXiv)

Event cameras have recently been shown beneficial for practical vision tasks, such as action recognition, thanks to their high temporal resolution, power efficiency, and reduced privacy concerns. However, current research is hindered by 1) the difficulty in processing events because of their prolonged duration and dynamic actions with complex and ambiguous semantics and 2) the redundant action depiction of the event frame representation with fixed stacks. We find language naturally conveys abundant semantic information, rendering it stunningly superior in reducing semantic uncertainty. In light of this, we propose ExACT, a novel approach that, for the first time, tackles event-based action recognition from a cross-modal conceptualizing perspective. Our ExACT brings two technical contributions. Firstly, we propose an adaptive fine-grained event (AFE) representation to adaptively filter out the repeated events for the stationary objects while preserving dynamic ones. This subtly enhances the performance of ExACT without extra computational cost. Then, we propose a conceptual reasoning-based uncertainty estimation module, which simulates the recognition process to enrich the semantic representation. In particular, conceptual reasoning builds the temporal relation based on the action semantics, and uncertainty estimation tackles the semantic uncertainty of actions based on the distributional representation. Experiments show that our ExACT achieves superior recognition accuracy of 94.83%(+2.23%), 90.10%(+37.47%) and 67.24% on PAF, HARDVS and our SeAct datasets respectively.