# What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging

arXiv: https://arxiv.org/abs/2304.13651 · HTML: https://mlsanity.com/paper/2304.13651/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging](https://mlsanity.com/paper/1904.06830/) (2019)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [Human Motion Prediction via Spatio-Temporal Inpainting](https://mlsanity.com/paper/1812.05478/) (2018)
- [Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation](https://mlsanity.com/paper/2004.03686/) (2020)
- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2204.04120/) (2022)
- [The Pose Knows: Video Forecasting by Generating Pose Futures](https://mlsanity.com/paper/1705.00053/) (2017)
- [PST900: RGB-Thermal Calibration, Dataset and Segmentation Network](https://mlsanity.com/paper/1909.10980/) (2019)
- [Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments](https://mlsanity.com/paper/1903.05690/) (2019)
- [Scene-aware Generative Network for Human Motion Synthesis](https://mlsanity.com/paper/2105.14804/) (2021)
- [Multi-Modal Fusion for End-to-End RGB-T Tracking](https://mlsanity.com/paper/1908.11714/) (2019)
- [Photo Wake-Up: 3D Character Animation from a Single Photo](https://mlsanity.com/paper/1812.02246/) (2018)
- [Predicting 3D Human Dynamics from Video](https://mlsanity.com/paper/1908.04781/) (2019)
- [Jointly Modeling Motion and Appearance Cues for Robust RGB-T Tracking](https://mlsanity.com/paper/2007.02041/) (2020)
- [Spatio-Temporal Gating-Adjacency GCN for Human Motion Prediction](https://mlsanity.com/paper/2203.01474/) (2022)
- [FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation](https://mlsanity.com/paper/2110.08988/) (2021)
- [Edge-aware Guidance Fusion Network for RGB Thermal Scene Parsing](https://mlsanity.com/paper/2112.05144/) (2021)
- [HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images](https://mlsanity.com/paper/2003.04645/) (2020)
- [Glass Segmentation with RGB-Thermal Image Pairs](https://mlsanity.com/paper/2204.05453/) (2022)
- [Forecasting Human Dynamics from Static Images](https://mlsanity.com/paper/1704.03432/) (2017)
- [Siamese Infrared and Visible Light Fusion Network for RGB-T Tracking](https://mlsanity.com/paper/2103.07302/) (2021)

## Built on by (most cited first, 8 of 8)

- [CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion](https://mlsanity.com/paper/2402.05889/) (2024)
- [Self-Supervised Spatial Correspondence Across Modalities](https://mlsanity.com/paper/2506.03148/) (2025)
- [An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding](https://mlsanity.com/paper/2505.01743/) (2025)
- [See the past: Time-Reversed Scene Reconstruction from Thermal Traces Using Visual Language Models](https://mlsanity.com/paper/2510.05408/) (2025)
- [3M-TI: High-Quality Mobile Thermal Imaging via Calibration-free Multi-Camera Cross-Modal Diffusion](https://mlsanity.com/paper/2511.19117/) (2025)
- [Time-Reversed Imaging: A Multimodal Benchmark and Framework for Reconstructing Past Human-Environment Interactions](https://mlsanity.com/paper/2607.22352/) (2026)
- [BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion](https://mlsanity.com/paper/2607.24110/) (2026)
- [Towards Privacy-Preserving Thermal Human Perception from Dataset to Deployment](https://mlsanity.com/paper/2610.14846/) (2026)

## Abstract (arXiv)

Inferring past human motion from RGB images is challenging due to the inherent uncertainty of the prediction problem. Thermal images, on the other hand, encode traces of past human-object interactions left in the environment via thermal radiation measurement. Based on this observation, we collect the first RGB-Thermal dataset for human motion analysis, dubbed Thermal-IM. Then we develop a three-stage neural network model for accurate past human pose estimation. Comprehensive experiments show that thermal cues significantly reduce the ambiguities of this task, and the proposed model achieves remarkable performance. The dataset is available at https://github.com/ZitianTang/Thermal-IM.