# Autoregressive Drillhole Modelling Under Distribution Shift

arXiv: https://arxiv.org/abs/2610.01204 · HTML: https://mlsanity.com/paper/2610.01204/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 30 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [TimeGPT-1](https://mlsanity.com/paper/2310.03589/) (2023)
- [TerraFM: A Scalable Foundation Model for Unified Multisensor Earth Observation](https://mlsanity.com/paper/2506.06281/) (2025)
- [Importance of spatial predictor variable selection in machine learning applications -- Moving from data reproduction to spatial prediction](https://mlsanity.com/paper/1908.07805/) (2019)
- [REOBench: Benchmarking Robustness of Earth Observation Foundation Models](https://mlsanity.com/paper/2505.16793/) (2025)
- [GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data](https://mlsanity.com/paper/2509.26016/) (2025)
- [UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation](https://mlsanity.com/paper/2606.23503/) (2026)
- [Classification of Geological Borehole Descriptions Using a Domain Adapted Large Language Model](https://mlsanity.com/paper/2407.10991/) (2024)
- [WLFM: A Well-Logs Foundation Model for Multi-Task and Cross-Well Geological Interpretation](https://mlsanity.com/paper/2509.18152/) (2025)

## Abstract (arXiv)

Autoregressive modelling has achieved remarkable success in language and sequence tasks by learning to predict future states from previous observation. Mineral-exploration drillholes provide a natural but largely unexplored setting for this paradigm: as drilling proceeds, lithology is revealed sequentially from shallow to deep, making prediction of deeper strata inherently autoregressive. Existing drillhole modelling, however, is dominated by spatial interpolation and reconstruction, or largely rely on masked modelling, leaving strictly autoregressive prediction largely underexplored. We introduce DrillBench, a benchmark of 49,671 Western Australian drillholes for next-layer prediction and autoregressive stratigraphic generation across a graded transfer spectrum, from local prediction through spatial shift to cross geological province transfer. Benchmarking classical, geostatistical, and neural models reveals a clear transfer boundary: spatial and geochemical conditioning provides large local gains but deteriorates sharply under stronger shift, whereas lithology-sequence autoregressive models transfer more robustly. Guided by this finding, we develop a backbone-agnostic recipe combining large-scale pretraining on historical drillholes with spatial retrieval of neighbouring lithology. Retrieval is most effective in weathered cover, when local spatial continuity remains informative, whereas pretraining contributes more strongly in bedrock and under broader geological shift. Together, they retain strong local performance while improving generalisation under spatial and cross-province shift, most markedly on the most distant splits. The benchmark and code are available at https://github.com/yihaoding/drillbench.