# WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments

arXiv: https://arxiv.org/abs/2312.15364 · HTML: https://mlsanity.com/paper/2312.15364/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 62 references that are in the map.
- First posted on arXiv 23 Dec 2023.

## Builds on (most cited first, 30 of 33)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving](https://mlsanity.com/paper/2003.03653/) (2020)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [Spherical Transformer for LiDAR-based 3D Recognition](https://mlsanity.com/paper/2303.12766/) (2023)
- [SemanticPOSS: A Point Cloud Dataset with Large Quantity of Dynamic Instances](https://mlsanity.com/paper/2002.09147/) (2020)
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](https://mlsanity.com/paper/2011.12954/) (2020)
- [Rope3D: TheRoadside Perception Dataset for Autonomous Driving and Monocular 3D Object Detection Task](https://mlsanity.com/paper/2203.13608/) (2022)
- [Boreas: A Multi-Season Autonomous Driving Dataset](https://mlsanity.com/paper/2203.10168/) (2022)
- [TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models](https://mlsanity.com/paper/2205.01791/) (2022)
- [3D Semantic Segmentation in the Wild: Learning Generalized Models for Adverse-Condition Point Clouds](https://mlsanity.com/paper/2304.00690/) (2023)
- [FuseSeg: LiDAR Point Cloud Segmentation Fusing Multi-Modal Data](https://mlsanity.com/paper/1912.08487/) (2019)
- [ORFD: A Dataset and Benchmark for Off-Road Freespace Detection](https://mlsanity.com/paper/2206.09907/) (2022)
- [Efficient Continuous-Time SLAM for 3D Lidar-Based Online Mapping](https://mlsanity.com/paper/1810.06802/) (2018)
- [Spatiotemporal Self-supervised Learning for Point Clouds in the Wild](https://mlsanity.com/paper/2303.16235/) (2023)
- [A Fine-Grained Dataset and its Efficient Semantic Segmentation for Unstructured Driving Scenarios](https://mlsanity.com/paper/2103.13109/) (2021)
- [Uncertainty-aware Panoptic Segmentation](https://mlsanity.com/paper/2206.14554/) (2022)
- [Wildcat: Online Continuous-Time 3D Lidar-Inertial SLAM](https://mlsanity.com/paper/2205.12595/) (2022)
- [Walking Your LiDOG: A Journey Through Multiple Domains for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2304.11705/) (2023)
- [IDD-3D: Indian Driving Dataset for 3D Unstructured Road Scenes](https://mlsanity.com/paper/2210.12878/) (2022)
- [Elasticity Meets Continuous-Time: Map-Centric Dense 3D LiDAR SLAM](https://mlsanity.com/paper/2008.02274/) (2020)

## Built on by (most cited first, 17 of 17)

- [TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation](https://mlsanity.com/paper/2505.10696/) (2025)
- [UAVScenes: A Multi-Modal Dataset for UAVs](https://mlsanity.com/paper/2507.22412/) (2025)
- [Advancing Off-Road Autonomous Driving: The Large-Scale ORAD-3D Dataset and Comprehensive Benchmarks](https://mlsanity.com/paper/2510.16500/) (2025)
- [Autonomous Driving in Unstructured Environments: How Far Have We Come?](https://mlsanity.com/paper/2410.07701/) (2024)
- [WildCross: A Cross-Modal Large Scale Benchmark for Place Recognition and Metric Depth Estimation in Natural Environments](https://mlsanity.com/paper/2603.01475/) (2026)
- [ROVER: A Multi-Season Dataset for Visual SLAM](https://mlsanity.com/paper/2412.02506/) (2024)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [ROVER: Robust Loop Closure Verification with Trajectory Prior in Repetitive Environments](https://mlsanity.com/paper/2508.13488/) (2025)
- [TOMD: A Trail-based Off-road Multimodal Dataset for Traversable Pathway Segmentation under Challenging Illumination Conditions](https://mlsanity.com/paper/2506.21630/) (2025)
- [TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation](https://mlsanity.com/paper/2607.10762/) (2026)
- [SydneyScapes: Image Segmentation for Australian Environments](https://mlsanity.com/paper/2504.07542/) (2025)
- [SilvaScenes: Tree Detection and Species Classification from Under-Canopy Images in Natural Forests](https://mlsanity.com/paper/2510.09458/) (2025)
- [Ilov3Splat: Instance-Level Open-Vocabulary 3D Scene Understanding in Gaussian Splatting](https://mlsanity.com/paper/2605.04506/) (2026)
- [SeasonScapes: Learning Large-scale Re-lightable 3D Landscapes with Seasonal Variation from Sparse Webcams](https://mlsanity.com/paper/2605.09039/) (2026)
- [Cross-Modal Benchmarking for Robotic Perception in Natural Environments](https://mlsanity.com/paper/2606.11563/) (2026)
- [ProGuT: Label-Efficient Panoptic Segmentation for Forest Scenes](https://mlsanity.com/paper/2609.36891/) (2026)
- [From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction](https://mlsanity.com/paper/2610.17630/) (2026)

## Abstract (arXiv)

Recent progress in semantic scene understanding has primarily been enabled by the availability of semantically annotated bi-modal (camera and LiDAR) datasets in urban environments. However, such annotated datasets are also needed for natural, unstructured environments to enable semantic perception for applications, including conservation, search and rescue, environment monitoring, and agricultural automation. Therefore, we introduce WildScenes, a bi-modal benchmark dataset consisting of multiple large-scale, sequential traversals in natural environments, including semantic annotations in high-resolution 2D images and dense 3D LiDAR point clouds, and accurate 6-DoF pose information. The data is (1) trajectory-centric with accurate localization and globally aligned point clouds, (2) calibrated and synchronized to support bi-modal training and inference, and (3) containing different natural environments over 6 months to support research on domain adaptation. Our 3D semantic labels are obtained via an efficient, automated process that transfers the human-annotated 2D labels from multiple views into 3D point cloud sequences, thus circumventing the need for expensive and time-consuming human annotation in 3D. We introduce benchmarks on 2D and 3D semantic segmentation and evaluate a variety of recent deep-learning techniques to demonstrate the challenges in semantic segmentation in natural environments. We propose train-val-test splits for standard benchmarks as well as domain adaptation benchmarks and utilize an automated split generation technique to ensure the balance of class label distributions. The WildScenes benchmark webpage is https://csiro-robotics.github.io/WildScenes, and the data is publicly available at https://data.csiro.au/collection/csiro:61541 .