# TreeFormers -- An Exploration of Vision Transformers for Deforestation Driver Classification

arXiv: https://arxiv.org/abs/2405.15989 · HTML: https://mlsanity.com/paper/2405.15989/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 18 references that are in the map.
- First posted on arXiv 25 May 2024.

## Builds on (most cited first, 8 of 8)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [ForestNet: Classifying Drivers of Deforestation in Indonesia using Deep Learning on Satellite Imagery](https://mlsanity.com/paper/2011.05479/) (2020)
- [Rotation Equivariant Deforestation Segmentation and Driver Classification](https://mlsanity.com/paper/2110.13097/) (2021)
- [SuperCon: Supervised Contrastive Learning for Imbalanced Skin Lesion Classification](https://mlsanity.com/paper/2202.05685/) (2022)
- [Multimodal SuperCon: Classifier for Drivers of Deforestation in Indonesia](https://mlsanity.com/paper/2207.14656/) (2022)

## Abstract (arXiv)

This paper addresses the critical issue of deforestation by exploring the application of vision transformers (ViTs) for classifying the drivers of deforestation using satellite imagery from Indonesian forests. Motivated by the urgency of this problem, I propose an approach that leverages ViTs and machine learning techniques. The input to my algorithm is a 332x332-pixel satellite image, and I employ a ViT architecture to predict the deforestation driver class; grassland shrubland, other, plantation, or smallholder agriculture. My methodology involves fine-tuning a pre-trained ViT on a dataset from the Stanford ML Group, and I experiment with rotational data augmentation techniques (among others) and embedding of longitudinal data to improve classification accuracy. I also tried training a ViT from scratch. Results indicate a significant improvement over baseline models, achieving a test accuracy of 72.9%. I conduct a comprehensive analysis, including error patterns and metrics, to highlight the strengths and limitations of my approach. This research contributes to the ongoing efforts to address deforestation challenges through advanced computer vision techniques.