# FATE: Pillar Encoding and Frequency-Aware Training for Event-Based Object Detection

arXiv: https://arxiv.org/abs/2606.17334 · HTML: https://mlsanity.com/paper/2606.17334/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Builds on (most cited first, 30 of 35)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [End-to-End Semi-Supervised Object Detection with Soft Teacher](https://mlsanity.com/paper/2106.09018/) (2021)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [Events-to-Video: Bringing Modern Computer Vision to Event Cameras](https://mlsanity.com/paper/1904.08298/) (2019)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [Recurrent Vision Transformers for Object Detection with Event Cameras](https://mlsanity.com/paper/2212.05598/) (2022)
- [A Large Scale Event-based Detection Dataset for Automotive](https://mlsanity.com/paper/2001.08499/) (2020)
- [Deep Directly-Trained Spiking Neural Networks for Object Detection](https://mlsanity.com/paper/2307.11411/) (2023)
- [State Space Models for Event Cameras](https://mlsanity.com/paper/2402.15584/) (2024)
- [Time-Ordered Recent Event (TORE) Volumes for Event Cameras](https://mlsanity.com/paper/2103.06108/) (2021)
- [GET: Group Event Transformer for Event-Based Vision](https://mlsanity.com/paper/2310.02642/) (2023)
- [EventNet: Asynchronous Recursive Event Processing](https://mlsanity.com/paper/1812.07045/) (2018)
- [SODFormer: Streaming Object Detection with Transformer Using Events and Frames](https://mlsanity.com/paper/2308.04047/) (2023)
- [Scene Adaptive Sparse Transformer for Event-based Object Detection](https://mlsanity.com/paper/2404.01882/) (2024)
- [A Voxel Graph CNN for Object Classification with Event Cameras](https://mlsanity.com/paper/2106.00216/) (2021)
- [Object Detection using Event Camera: A MoE Heat Conduction based Detector and A New Benchmark Dataset](https://mlsanity.com/paper/2412.06647/) (2024)
- [LEOD: Label-Efficient Object Detection for Event Cameras](https://mlsanity.com/paper/2311.17286/) (2023)
- [SMamba: Sparse Mamba for Event-based Object Detection](https://mlsanity.com/paper/2501.11971/) (2025)
- [A Hybrid ANN-SNN Architecture for Low-Power and Low-Latency Visual Perception](https://mlsanity.com/paper/2303.14176/) (2023)
- [FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies](https://mlsanity.com/paper/2412.06708/) (2024)
- [EvRT-DETR: Latent Space Adaptation of Image Detectors for Event-based Vision](https://mlsanity.com/paper/2412.02890/) (2024)
- [ALERT-Transformer: Bridging Asynchronous and Synchronous Machine Learning for Real-Time Event-based Spatio-Temporal Data](https://mlsanity.com/paper/2402.01393/) (2024)
- [PLEIADES: Building Temporal Kernels with Orthogonal Polynomials](https://mlsanity.com/paper/2405.12179/) (2024)

## Abstract (arXiv)

Event cameras are bio-inspired sensors that asynchronously capture logarithmic intensity changes, offering inherent advantages in high-speed and high-dynamic-range scenarios. However, the sparse and asynchronous nature of event streams poses a fundamental challenge for modern deep learning architectures. To enable compatibility with standard models, most existing approaches partition the accumulation window into fixed temporal sub-bins. While effective for spatial processing, this internal discretization discards fine-grained temporal structure and constrains inference to the low temporal frequencies imposed by training supervision. To address this limitation, we propose FATE, a unified framework built upon a novel Pillar Encoding (PE). While operating over discrete macro-accumulation windows dictated by the target frequency, PE avoids internal temporal sub-binning. It organizes events into spatial pillars and approximates their intra-window evolution via projection onto a continuous-time orthogonal polynomial basis. This formulation yields an L2-optimal representation that retains rich temporal dynamics in a dense pseudo-image, mitigating information loss under sparse event conditions. To fully leverage this representation, we introduce Frequency-Aware Training (FAT), a soft mean-teacher curriculum that generates temporally dense pseudo-labels, effectively bridging the mismatch between low-frequency supervision and high-frequency inference. Extensive experiments demonstrate that FATE generalizes across architectural paradigms and consistently outperforms strong baselines. It enables robust object detection at high temporal resolutions up to 200 Hz, while incurring minimal overhead in parameter count and inference latency