# E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning

arXiv: https://arxiv.org/abs/2311.18433 · HTML: https://mlsanity.com/paper/2311.18433/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades](https://mlsanity.com/paper/1507.07629/) (2015)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [Deep Learning in Spiking Neural Networks](https://mlsanity.com/paper/1804.08150/) (2018)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM](https://mlsanity.com/paper/1610.08336/) (2016)
- [Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars](https://mlsanity.com/paper/1804.01310/) (2018)
- [RPM-Net: Robust Point Matching using Learned Features](https://mlsanity.com/paper/2003.13479/) (2020)
- [Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy](https://mlsanity.com/paper/2009.08283/) (2020)
- [AEGNN: Asynchronous Event-based Graph Neural Networks](https://mlsanity.com/paper/2203.17149/) (2022)
- [Event-based Asynchronous Sparse Convolutional Networks](https://mlsanity.com/paper/2003.09148/) (2020)
- [Time-Ordered Recent Event (TORE) Volumes for Event Cameras](https://mlsanity.com/paper/2103.06108/) (2021)
- [2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud](https://mlsanity.com/paper/1904.09742/) (2019)
- [Learning Dense and Continuous Optical Flow from an Event Camera](https://mlsanity.com/paper/2211.09078/) (2022)
- [DeepI2P: Image-to-Point Cloud Registration via Deep Classification](https://mlsanity.com/paper/2104.03501/) (2021)
- [Graph-based Asynchronous Event Processing for Rapid Object Recognition](https://mlsanity.com/paper/2308.14419/) (2023)
- [Event-based Motion Segmentation with Spatio-Temporal Graph Cuts](https://mlsanity.com/paper/2012.08730/) (2020)
- [EV-IMO: Motion Segmentation Dataset and Learning Pipeline for Event Cameras](https://mlsanity.com/paper/1903.07520/) (2019)
- [VECtor: A Versatile Event-Centric Benchmark for Multi-Sensor SLAM](https://mlsanity.com/paper/2207.01404/) (2022)
- [P2-Net: Joint Description and Detection of Local Features for Pixel and Point Matching](https://mlsanity.com/paper/2103.01055/) (2021)
- [STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Prediction](https://mlsanity.com/paper/2203.16084/) (2022)

## Built on by (most cited first, 11 of 11)

- [RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method](https://mlsanity.com/paper/2403.19501/) (2024)
- [A Survey of 3D Reconstruction with Event Cameras](https://mlsanity.com/paper/2505.08438/) (2025)
- [Event2Vec: Processing Neuromorphic Events Directly by Representations in Vector Space](https://mlsanity.com/paper/2504.15371/) (2025)
- [Event Camera Guided Visual Media Restoration & 3D Reconstruction: A Survey](https://mlsanity.com/paper/2509.09971/) (2025)
- [EventFlash: Towards Efficient MLLMs for Event-Based Vision](https://mlsanity.com/paper/2602.03230/) (2026)
- [A Survey on Event-driven 3D Reconstruction: Development under Different Categories](https://mlsanity.com/paper/2503.19753/) (2025)
- [EvRainDrop: HyperGraph-guided Completion for Effective Frame and Event Stream Aggregation](https://mlsanity.com/paper/2511.21439/) (2025)
- [Learning to Remove Lens Flare in Event Camera](https://mlsanity.com/paper/2512.09016/) (2025)
- [OmniEvent: Unified Event Representation Learning](https://mlsanity.com/paper/2508.01842/) (2025)
- [LEAR: Learning Edge-Aware Representations for Event-to-LiDAR Localization](https://mlsanity.com/paper/2603.01839/) (2026)
- [VGER: Voxel-Guided Global Event Ranking for Event Cloud Attribution](https://mlsanity.com/paper/2608.01470/) (2026)

## Abstract (arXiv)

Event cameras have emerged as a promising vision sensor in recent years due to their unparalleled temporal resolution and dynamic range. While registration of 2D RGB images to 3D point clouds is a long-standing problem in computer vision, no prior work studies 2D-3D registration for event cameras. To this end, we propose E2PNet, the first learning-based method for event-to-point cloud registration. The core of E2PNet is a novel feature representation network called Event-Points-to-Tensor (EP2T), which encodes event data into a 2D grid-shaped feature tensor. This grid-shaped feature enables matured RGB-based frameworks to be easily used for event-to-point cloud registration, without changing hyper-parameters and the training procedure. EP2T treats the event input as spatio-temporal point clouds. Unlike standard 3D learning architectures that treat all dimensions of point clouds equally, the novel sampling and information aggregation modules in EP2T are designed to handle the inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC and VECtor datasets demonstrate the superiority of E2PNet over hand-crafted and other learning-based methods. Compared to RGB-based registration, E2PNet is more robust to extreme illumination or fast motion due to the use of event data. Beyond 2D-3D registration, we also show the potential of EP2T for other vision tasks such as flow estimation, event-to-image reconstruction and object recognition. The source code can be found at: https://github.com/Xmu-qcj/E2PNet.