mlsanityData to 7 Oct 2026

Tools for Extracting Spatio-Temporal Patterns in Meteorological Image Sequences: From Feature Engineering to Attention-Based Neural Networks

arXiv 2210.12310 ↗First posted 22 Oct 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Atmospheric processes involve both space and time. This is why human analysis of atmospheric imagery can often extract more information from animated loops of image sequences than from individual images. Automating such an analysis requires the ability to identify spatio-temporal patterns in image sequences which is a very challenging task, because of the endless possibilities of patterns in both space and time. In this paper we review different concepts and techniques that are useful to extract spatio-temporal context specifically for meteorological applications. In this survey we first motivate the need for these approaches in meteorology using two applications, solar forecasting and detecting convection from satellite imagery. Then we provide an overview of many different concepts and techniques that are helpful for the interpretation of meteorological image sequences, such as (1) feature engineering methods to strengthen the desired signal in the input, using meteorological knowledge, classic image processing, harmonic analysis and topological data analysis (2) explain how different convolution filters (2D/3D/LSTM-convolution) can be utilized strategically in convolutional neural network architectures to find patterns in both space and time (3) discuss the powerful new concept of 'attention' in neural networks and the powerful abilities it brings to the interpretation of image sequences (4) briefly survey strategies from unsupervised, self-supervised and transfer learning to reduce the need for large labeled datasets. We hope that presenting an overview of these tools - many of which are underutilized - will help accelerate progress in this area.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 131 references that are in the map. First posted on arXiv 22 Oct 2022.

Builds on

49 of 131 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  9. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  11. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  12. 2015Spatial Transformer Networkscited by 1,423
37 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  2. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  3. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  4. 2014Neural Turing Machinescited by 630
  5. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 348
  6. 2021Skillful Precipitation Nowcasting using Deep Generative Models of Radarcited by 308
  7. 2021Remote Sensing Image Change Detection with Transformerscited by 305
  8. 2019Tackling Climate Change with Machine Learningcited by 264
  9. 2019Time2Vec: Learning a Vector Representation of Timecited by 208
  10. 2020Geography-Aware Self-Supervised Learningcited by 190
  11. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 146
  12. 2021Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networkscited by 141
  13. 2020MetNet: A Neural Weather Model for Precipitation Forecastingcited by 128
  14. 2018Tile2Vec: Unsupervised representation learning for spatially distributed datacited by 109
  15. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  16. 2020SmaAt-UNet: Precipitation Nowcasting using a Small Attention-UNet Architecturecited by 71
  17. 2020The color out of space: learning self-supervised representations for Earth Observation imagerycited by 34
  18. 2018Deep Photovoltaic Nowcastingcited by 21
  19. 2021Skillful Twelve Hour Precipitation Forecasts using Large Context Neural Networkscited by 20
  20. 2020Physics-Informed Neural Network Super Resolution for Advection-Diffusion Modelscited by 16
  21. 2021Manipulation Detection in Satellite Images Using Vision Transformercited by 16
  22. 2020Convolutional Neural Networks applied to sky images for short-term solar irradiance forecastingcited by 12
  23. 2020An Attention-Based System for Damage Assessment Using Satellite Imagerycited by 11
  24. 2017DeepRain: ConvLSTM Network for Precipitation Prediction using Multichannel Radar Datacited by 10
  25. 2019A deep learning approach to solar-irradiance forecasting in sky-videoscited by 10
  26. 2021Nowcasting-Nets: Deep Neural Network Structures for Precipitation Nowcasting Using IMERGcited by 10
  27. 2021Deep Transfer Learning for Land Use and Land Cover Classification: A Comparative Studycited by 9
  28. 2022CrowdFormer: Weakly-supervised Crowd counting with Improved Generalizabilitycited by 8
  29. 2019Short-term forecasting of solar irradiance without local telemetry: a generalized model using satellite datacited by 7
  30. 2021Attention-augmented Spatio-Temporal Segmentation for Land Cover Mappingcited by 7
  31. 2022A Primer on Topological Data Analysis to Support Image Analysis Tasks in Environmental Sciencecited by 7
  32. 2020Generative Modeling for Atmospheric Convectioncited by 5
  33. 2018An Attention-Based Approach for Single Image Super Resolutioncited by 3
  34. 2019Using Persistent Homology to Quantify a Diurnal Cycle in Hurricane Felixcited by 3
  35. 2021A Moment in the Sun: Solar Nowcasting from Multispectral Satellite Data using Self-Supervised Learningcited by 3
  36. 2022Can we integrate spatial verification methods into neural-network loss functions for atmospheric science?cited by 3
  37. 2022Trainable Wavelet Neural Network for Non-Stationary Signalscited by 2

Built on by

Not cited yet.