SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation
Not on the map. Why?
Abstract (arXiv)
Event cameras generate asynchronous and sparse event streams capturing changes in light intensity. They offer significant advantages over conventional frame-based cameras, such as a higher dynamic range and an extremely faster data rate, making them particularly useful in scenarios involving fast motion or challenging lighting conditions. Spiking neural networks (SNNs) share similar asynchronous and sparse characteristics and are well-suited for processing data from event cameras. Inspired by the potential of transformers and spike-driven transformers (spikeformers) in other computer vision tasks, we propose two solutions for fast and robust optical flow estimation for event cameras: STTFlowNet and SDformerFlow. STTFlowNet adopts a U-shaped artificial neural network (ANN) architecture with spatiotemporal shifted window self-attention (swin) transformer encoders, while SDformerFlow presents its fully spiking counterpart, incorporating swin spikeformer encoders. Furthermore, we present two variants of the spiking version with different neuron models. Our work is the first to make use of spikeformers for dense optical flow estimation. We conduct end-to-end training for all models using supervised learning. Our results yield state-of-the-art performance among SNN-based event optical flow methods on both the DSEC and MVSEC datasets, and show significant reduction in power consumption compared to the equivalent ANNs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 47 of its 57 references that are in the map. First posted on arXiv 6 Sep 2024.
Builds on
47 of 57 references.
References · 47
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
- 2021Video Swin Transformercited by 1,090
- 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 604
- 2021Deep Residual Learning in Spiking Neural Networkscited by 326
- 2021GMFlow: Learning Optical Flow via Global Matchingcited by 324
- 2018Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotioncited by 304
- 2021DSEC: A Stereo Event Camera Dataset for Driving Scenarioscited by 297
- 2022FlowFormer: A Transformer Architecture for Optical Flowcited by 273
- 2022Spikformer: When Spiking Neural Network Meets Transformercited by 263
35 more
- 2018The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perceptioncited by 251
- 2018EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Camerascited by 223
- 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 163
- 2021E-RAFT: Dense Optical Flow from Event Camerascited by 149
- 2023Spike-driven Transformercited by 141
- 2022MetaFormer Baselines for Visioncited by 131
- 2022Attention Spiking Neural Networkscited by 103
- 2021Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networkscited by 99
- 2020Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networkscited by 98
- 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
- 2022Secrets of Event-Based Optical Flowcited by 92
- 2022CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flowcited by 82
- 2021Advancing Spiking Neural Networks towards Deep Residual Learningcited by 76
- 2022Multi-Frame Self-Supervised Depth with Transformerscited by 64
- 2022GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networkscited by 63
- 2022Learning Dense and Continuous Optical Flow from an Event Cameracited by 59
- 2021Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimationcited by 50
- 2023Event Camera Data Pre-trainingcited by 49
- 2024QKFormer: Hierarchical Spiking Transformer using Q-K Attentioncited by 46
- 2023Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependenciescited by 41
- 2023Taming Contrast Maximization for Learning Sequential, Low-latency, Event-based Optical Flowcited by 39
- 2022Masked Spiking Transformercited by 38
- 2024SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networkscited by 38
- 2022ResFormer: Scaling ViTs with Multi-Resolution Trainingcited by 34
- 2023Optical flow estimation from event-based cameras and spiking neural networkscited by 33
- 2023BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimationcited by 33
- 2023TransFlow: Transformer as Flow Learnercited by 32
- 2023TMA: Temporal Motion Aggregation for Event-based Optical Flowcited by 31
- 2023Learning Optical Flow from Event Camera with Rendered Datasetcited by 26
- 2024ViTAR: Vision Transformer with Any Resolutioncited by 25
- 2022Adaptive-SpikeNet: Event-based Optical Flow Estimation using Spiking Neural Networks with Learnable Neuronal Dynamicscited by 19
- 2022Event-based Temporally Dense Optical Flow Estimation with Sequential Learningcited by 15
- 2023Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformercited by 10
- 2024SVFormer: A Direct Training Spiking Transformer for Efficient Video Action Recognitioncited by 3
- 2024A Novel Spike Transformer Network for Depth Estimation from Event Cameras via Cross-modality Knowledge Distillationcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.