# EvMic: Event-based Non-contact sound recovery from effective spatial-temporal modeling

arXiv: https://arxiv.org/abs/2504.02402 · HTML: https://mlsanity.com/paper/2504.02402/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 54 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [State Space Models for Event Cameras](https://mlsanity.com/paper/2402.15584/) (2024)
- [Learning Dense and Continuous Optical Flow from an Event Camera](https://mlsanity.com/paper/2211.09078/) (2022)
- [Learning-based Video Motion Magnification](https://mlsanity.com/paper/1804.02684/) (2018)
- [FullSubNet+: Channel Attention FullSubNet with Complex Spectrograms for Speech Enhancement](https://mlsanity.com/paper/2203.12188/) (2022)
- [EAS-SNN: End-to-End Adaptive Sampling and Representation for Event-based Detection with Recurrent Spiking Neural Networks](https://mlsanity.com/paper/2403.12574/) (2024)
- [Frequency Cam: Imaging Periodic Signals in Real-Time](https://mlsanity.com/paper/2211.00198/) (2022)
- [SpikMamba: When SNN meets Mamba in Event-based Human Action Recognition](https://mlsanity.com/paper/2410.16746/) (2024)
- [Event-Based Motion Magnification](https://mlsanity.com/paper/2402.11957/) (2024)
- [EvSign: Sign Language Recognition and Translation with Streaming Events](https://mlsanity.com/paper/2407.12593/) (2024)
- [Non-Invasive Qualitative Vibration Analysis using Event Camera](https://mlsanity.com/paper/2410.14364/) (2024)

## Abstract (arXiv)

When sound waves hit an object, they induce vibrations that produce high-frequency and subtle visual changes, which can be used for recovering the sound. Early studies always encounter trade-offs related to sampling rate, bandwidth, field of view, and the simplicity of the optical path. Recent advances in event camera hardware show good potential for its application in visual sound recovery, because of its superior ability in capturing high-frequency signals. However, existing event-based vibration recovery methods are still sub-optimal for sound recovery. In this work, we propose a novel pipeline for non-contact sound recovery, fully utilizing spatial-temporal information from the event stream. We first generate a large training set using a novel simulation pipeline. Then we designed a network that leverages the sparsity of events to capture spatial information and uses Mamba to model long-term temporal information. Lastly, we train a spatial aggregation block to aggregate information from different locations to further improve signal quality. To capture event signals caused by sound waves, we also designed an imaging system using a laser matrix to enhance the gradient and collected multiple data sequences for testing. Experimental results on synthetic and real-world data demonstrate the effectiveness of our method.