# Advanced Hybrid Transformer LSTM Technique with Attention and TS Mixer for Drilling Rate of Penetration Prediction

arXiv: https://arxiv.org/abs/2508.05210 · HTML: https://mlsanity.com/paper/2508.05210/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 80 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [TSMixer: An All-MLP Architecture for Time Series Forecasting](https://mlsanity.com/paper/2303.06053/) (2023)
- [A Recent Survey of Vision Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2312.00634/) (2023)
- [xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories](https://mlsanity.com/paper/2410.16928/) (2024)
- [A Comprehensive Survey on Architectural Advances in Deep CNNs: Challenges, Applications, and Emerging Research Directions](https://mlsanity.com/paper/2503.16546/) (2025)
- [Autonomous Drone Swarm Navigation and Multi-target Tracking in 3D Environments with Dynamic Obstacles](https://mlsanity.com/paper/2202.06253/) (2022)
- [RS-FME-SwinT: A Novel Feature Map Enhancement Framework Integrating Customized SwinT with Residual and Spatial CNN for Monkeypox Diagnosis](https://mlsanity.com/paper/2410.01216/) (2024)
- [GCT: Gated Contextual Transformer for Sequential Audio Tagging](https://mlsanity.com/paper/2210.12541/) (2022)
- [DTization: A New Method for Supervised Feature Scaling](https://mlsanity.com/paper/2404.17937/) (2024)

## Built on by (most cited first, 1 of 1)

- [MambaFormer: Token-Level Guided Routing Mixture-of-Experts for Accurate and Efficient Clinical Assistance](https://mlsanity.com/paper/2601.01260/) (2026)

## Abstract (arXiv)

Rate of Penetration (ROP) prediction is critical for drilling optimization yet remains challenging due to the nonlinear, dynamic, and heterogeneous characteristics of drilling data. Conventional empirical, physics-based, and standard machine learning models rely on oversimplified assumptions or intensive feature engineering, constraining their capacity to model long-term dependencies and intricate feature interactions. To address these issues, this study presents a new deep learning Hybrid LSTM-Trans-Mixer-Att framework that first processes input data through a customized Long Short-Term Memory (LSTM) network to capture multi-scale temporal dependencies aligned with drilling cycles. Subsequently, an Enhanced Transformer encoder with drilling-specific positional encodings and real-time optimization refines the features. Concurrently, a parallel Time-Series Mixer (TS-Mixer) block introduced facilitates efficient cross-feature interaction modeling of static and categorical parameters, including lithological indices and mud properties. The feature representations extracted from the Enhanced Transformer and TS-Mixer modules are integrated through a dedicated fusion layer. Finally, an adaptive attention mechanism then dynamically assigns contextual weights to salient features, enhancing discriminative representation learning and enabling high-fidelity ROP prediction. The proposed framework combines sequential memory, static feature interactions, global context learning, and dynamic feature weighting, providing a comprehensive solution for the heterogeneous and event-driven nature of drilling dynamics. Experimental validation on real-world drilling datasets demonstrates superior performance, achieving an Rsquare of 0.9991 and a MAPE of 1.447%, significantly outperforming existing baseline and hybrid models.