# QCATS: Query Context-Aware Transformer Slicing for Efficient Predictive Query Processing

arXiv: https://arxiv.org/abs/2610.09894 · HTML: https://mlsanity.com/paper/2610.09894/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 5 of its 54 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 5 of 5)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [FastMoE: A Fast Mixture-of-Expert Training System](https://mlsanity.com/paper/2103.13262/) (2021)
- [ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling](https://mlsanity.com/paper/2410.17954/) (2024)

## Abstract (arXiv)

In-database predictive query processing increasingly applies Transformer-based models within relational pipelines. However, existing in-database inference typically exposes only tuple-level model inputs to the inference runtime, leaving relational predicates and metadata statistics invisible to neural execution planning. In this paper, we propose QCATS, a query context-aware transformer slicing framework that enables efficient sparse inference inside database systems. QCATS executes at query granularity: instead of routing individual tokens or tuples during inference, it uses query predicates and metadata statistics to pre-select context-aligned FFN slices before model execution. The framework comprises offline expert construction and lightweight query-level routing that dynamically selects experts during execution. QCATS further introduces system optimizations, including asynchronous CPU-GPU pipelines and routing-aware batching. Experiments on four predictive-query workloads with BERT-base and Qwen-0.6B show that QCATS achieves up to 4.42x latency reduction while preserving prediction accuracy comparable to dense baselines.