# DeepGraviLens: a Multi-Modal Architecture for Classifying Gravitational Lensing Data

arXiv: https://arxiv.org/abs/2205.00701 · HTML: https://mlsanity.com/paper/2205.00701/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 128 references that are in the map.
- First posted on arXiv 2 May 2022.

## Builds on (most cited first, 17 of 17)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series](https://mlsanity.com/paper/1811.10166/) (2018)
- [Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge](https://mlsanity.com/paper/2012.12975/) (2020)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- [Capacity and Trainability in Recurrent Neural Networks](https://mlsanity.com/paper/1611.09913/) (2016)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning](https://mlsanity.com/paper/2106.02445/) (2021)
- [Multimodal Spatio-Temporal Deep Learning Approach for Neonatal Postoperative Pain Assessment](https://mlsanity.com/paper/2012.02175/) (2020)
- [DeepStroke: An Efficient Stroke Screening Framework for Emergency Rooms with Multimodal Adversarial Deep Learning](https://mlsanity.com/paper/2109.12065/) (2021)
- [Controlling for Confounders in Multimodal Emotion Classification via Adversarial Learning](https://mlsanity.com/paper/1908.08979/) (2019)
- [Memory Visualization for Gated Recurrent Neural Networks in Speech Recognition](https://mlsanity.com/paper/1609.08789/) (2016)
- [Inferring Black Hole Properties from Astronomical Multivariate Time Series with Bayesian Attentive Neural Processes](https://mlsanity.com/paper/2106.01450/) (2021)
- [Comparison of Multi-Class and Binary Classification Machine Learning Models in Identifying Strong Gravitational Lenses](https://mlsanity.com/paper/2002.11849/) (2020)

## Built on by (most cited first, 2 of 2)

- [Time Series Analysis in Compressor-Based Machines: A Survey](https://mlsanity.com/paper/2402.17802/) (2024)
- [DeepRed: an architecture for redshift estimation](https://mlsanity.com/paper/2602.11281/) (2026)

## Abstract (arXiv)

Gravitational lensing is the relativistic effect generated by massive bodies, which bend the space-time surrounding them. It is a deeply investigated topic in astrophysics and allows validating theoretical relativistic results and studying faint astrophysical objects that would not be visible otherwise. In recent years Machine Learning methods have been applied to support the analysis of the gravitational lensing phenomena by detecting lensing effects in data sets consisting of images associated with brightness variation time series. However, the state-of-art approaches either consider only images and neglect time-series data or achieve relatively low accuracy on the most difficult data sets. This paper introduces DeepGraviLens, a novel multi-modal network that classifies spatio-temporal data belonging to one non-lensed system type and three lensed system types. It surpasses the current state of the art accuracy results by ≈ 3% to ≈ 11%, depending on the considered data set. Such an improvement will enable the acceleration of the analysis of lensed objects in upcoming astrophysical surveys, which will exploit the petabytes of data collected, e.g., from the Vera C. Rubin Observatory.