# STFNets: Learning Sensing Signals from the Time-Frequency Perspective with Short-Time Fourier Neural Networks

arXiv: https://arxiv.org/abs/1902.07849 · HTML: https://mlsanity.com/paper/1902.07849/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2019.

## Built on by (most cited first, 23 of 23)

- [Deep Time Series Models: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2407.13278/) (2024)
- [RF-Diffusion: Radio Signal Generation via Time-Frequency Diffusion](https://mlsanity.com/paper/2404.09140/) (2024)
- [What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?](https://mlsanity.com/paper/2202.05998/) (2022)
- [Machine Learning for Microcontroller-Class Hardware: A Review](https://mlsanity.com/paper/2205.14550/) (2022)
- [Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals](https://mlsanity.com/paper/2309.05927/) (2023)
- [Time Series Analysis in Frequency Domain: A Survey of Open Challenges, Opportunities and Benchmarks](https://mlsanity.com/paper/2504.07099/) (2025)
- [RFBoost: Understanding and Boosting Deep WiFi Sensing via Physical Data Augmentation](https://mlsanity.com/paper/2410.07230/) (2024)
- [E2USD: Efficient-yet-effective Unsupervised State Detection for Multivariate Time Series](https://mlsanity.com/paper/2402.14041/) (2024)
- [A Joint Time-frequency Domain Transformer for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2305.14649/) (2023)
- [Phase-driven Domain Generalizable Learning for Nonstationary Time Series](https://mlsanity.com/paper/2402.05960/) (2024)
- [From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signals](https://mlsanity.com/paper/2405.16083/) (2024)
- [Context-Aware Predictive Coding: A Representation Learning Framework for WiFi Sensing](https://mlsanity.com/paper/2410.01825/) (2024)
- [SudokuSens: Enhancing Deep Learning Robustness for IoT Sensing Applications using a Generative Approach](https://mlsanity.com/paper/2402.02275/) (2024)
- [Robust Multimodal Fusion for Human Activity Recognition](https://mlsanity.com/paper/2303.04636/) (2023)
- [Rank Supervised Contrastive Learning for Time Series Classification](https://mlsanity.com/paper/2401.18057/) (2024)
- [NeurIT: Pushing the Limit of Neural Inertial Tracking for Indoor Robotic IoT](https://mlsanity.com/paper/2404.08939/) (2024)
- [Towards White-Box Deep Wireless Sensing](https://mlsanity.com/paper/2507.21799/) (2025)
- [Auto-Augmentation Contrastive Learning for Wearable-based Human Activity Recognition](https://mlsanity.com/paper/2602.02542/) (2026)
- [An Exploratory Study to Repurpose LLMs to a Unified Architecture for Time Series Classification](https://mlsanity.com/paper/2601.09971/) (2026)
- [RF-LEGO: Modularized Signal Processing-Deep Learning Co-Design for RF Sensing via Deep Unrolling](https://mlsanity.com/paper/2604.10183/) (2026)
- [Dywave: Event-Aligned Dynamic Tokenization for Heterogeneous IoT Sensing Signals](https://mlsanity.com/paper/2605.14014/) (2026)
- [Sound-based Multi-Person 3D Pose Estimation](https://mlsanity.com/paper/2609.04902/) (2026)
- [SpecRegMatch: Robust Semi-Supervised Regression for Vehicle Interior Noise Prediction](https://mlsanity.com/paper/2609.34111/) (2026)

## Abstract (arXiv)

Recent advances in deep learning motivate the use of deep neural networks in Internet-of-Things (IoT) applications. These networks are modelled after signal processing in the human brain, thereby leading to significant advantages at perceptual tasks such as vision and speech recognition. IoT applications, however, often measure physical phenomena, where the underlying physics (such as inertia, wireless signal propagation, or the natural frequency of oscillation) are fundamentally a function of signal frequencies, offering better features in the frequency domain. This observation leads to a fundamental question: For IoT applications, can one develop a new brand of neural network structures that synthesize features inspired not only by the biology of human perception but also by the fundamental nature of physics? Hence, in this paper, instead of using conventional building blocks (e.g., convolutional and recurrent layers), we propose a new foundational neural network building block, the Short-Time Fourier Neural Network (STFNet). It integrates a widely-used time-frequency analysis method, the Short-Time Fourier Transform, into data processing to learn features directly in the frequency domain, where the physics of underlying phenomena leave better foot-prints. STFNets bring additional flexibility to time-frequency analysis by offering novel nonlinear learnable operations that are spectral-compatible. Moreover, STFNets show that transforming signals to a domain that is more connected to the underlying physics greatly simplifies the learning process. We demonstrate the effectiveness of STFNets with extensive experiments. STFNets significantly outperform the state-of-the-art deep learning models in all experiments. A STFNet, therefore, demonstrates superior capability as the fundamental building block of deep neural networks for IoT applications for various sensor inputs.