# Transformer-based Multimodal Change Detection with Multitask Consistency Constraints

arXiv: https://arxiv.org/abs/2310.09276 · HTML: https://mlsanity.com/paper/2310.09276/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 71 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Fully Convolutional Siamese Networks for Change Detection](https://mlsanity.com/paper/1810.08462/) (2018)
- [A Transformer-Based Siamese Network for Change Detection](https://mlsanity.com/paper/2201.01293/) (2022)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation](https://mlsanity.com/paper/1906.03525/) (2019)
- [Learning to Compare Image Patches via Convolutional Neural Networks](https://mlsanity.com/paper/1504.03641/) (2015)
- [DynamicEarthNet: Daily Multi-Spectral Satellite Dataset for Semantic Change Segmentation](https://mlsanity.com/paper/2203.12560/) (2022)
- [The Edge of Depth: Explicit Constraints between Segmentation and Depth](https://mlsanity.com/paper/2004.00171/) (2020)
- [Deep Image Translation with an Affinity-Based Change Prior for Unsupervised Multimodal Change Detection](https://mlsanity.com/paper/2001.04271/) (2020)
- [Unsupervised Multimodal Change Detection Based on Structural Relationship Graph Representation Learning](https://mlsanity.com/paper/2210.00941/) (2022)
- [Inferring 3D change detection from bitemporal optical images](https://mlsanity.com/paper/2205.15903/) (2022)
- [Change Detection between Multimodal Remote Sensing Data Using Siamese CNN](https://mlsanity.com/paper/1807.09562/) (2018)
- [Deep Unsupervised Learning for 3D ALS Point Cloud Change Detection](https://mlsanity.com/paper/2305.03529/) (2023)
- [LSNet: Extremely Light-Weight Siamese Network For Change Detection in Remote Sensing Image](https://mlsanity.com/paper/2201.09156/) (2022)
- [Building change detection based on multi-scale filtering and grid partition](https://mlsanity.com/paper/1908.08164/) (2019)
- [Rapid Deforestation and Burned Area Detection using Deep Multimodal Learning on Satellite Imagery](https://mlsanity.com/paper/2307.04916/) (2023)

## Built on by (most cited first, 4 of 4)

- [Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model](https://mlsanity.com/paper/2510.11509/) (2025)
- [Change-Aware Siamese Network for Surface Defects Segmentation under Complex Background](https://mlsanity.com/paper/2409.00589/) (2024)
- [Prior-guided Fusion of Multimodal Features for Change Detection from Optical-SAR Images](https://mlsanity.com/paper/2604.05527/) (2026)
- [DPG-CD: Depth-Prior-Guided Cross-Modal Joint 2D-3D Change Detection](https://mlsanity.com/paper/2605.07151/) (2026)

## Abstract (arXiv)

Change detection plays a fundamental role in Earth observation for analyzing temporal iterations over time. However, recent studies have largely neglected the utilization of multimodal data that presents significant practical and technical advantages compared to single-modal approaches. This research focuses on leveraging pre-event digital surface model (DSM) data and post-event digital aerial images captured at different times for detecting change beyond 2D. We observe that the current change detection methods struggle with the multitask conflicts between semantic and height change detection tasks. To address this challenge, we propose an efficient Transformer-based network that learns shared representation between cross-dimensional inputs through cross-attention. It adopts a consistency constraint to establish the multimodal relationship. Initially, pseudo-changes are derived by employing height change thresholding. Subsequently, the L2 distance between semantic and pseudo-changes within their overlapping regions is minimized. This explicitly endows the height change detection (regression task) and semantic change detection (classification task) with representation consistency. A DSM-to-image multimodal dataset encompassing three cities in the Netherlands was constructed. It lays a new foundation for beyond-2D change detection from cross-dimensional inputs. Compared to five state-of-the-art change detection methods, our model demonstrates consistent multitask superiority in terms of semantic and height change detection. Furthermore, the consistency strategy can be seamlessly adapted to the other methods, yielding promising improvements.