# TLP: A Deep Learning-based Cost Model for Tensor Program Tuning arXiv: https://arxiv.org/abs/2211.03578 ยท HTML: https://mlsanity.com/paper/2211.03578/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 17 of its 43 references that are in the map. - First posted on arXiv 7 Nov 2022. ## Builds on (most cited first, 17 of 17) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017) - [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019) - [Ansor: Generating High-Performance Tensor Programs for Deep Learning](https://mlsanity.com/paper/2006.06762/) (2020) - [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016) - [Learning to Optimize Tensor Programs](https://mlsanity.com/paper/1805.08166/) (2018) - [Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions](https://mlsanity.com/paper/1802.04730/) (2018) - [TVM: An Automated End-to-End Optimizing Compiler for Deep Learning](https://mlsanity.com/paper/1802.04799/) (2018) - [Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code](https://mlsanity.com/paper/1804.10694/) (2018) - [A Deep Learning Based Cost Model for Automatic Code Optimization](https://mlsanity.com/paper/2104.04955/) (2021) - [Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation](https://mlsanity.com/paper/2001.08743/) (2020) - [Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning](https://mlsanity.com/paper/1801.08058/) (2018) - [Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance](https://mlsanity.com/paper/2110.15238/) (2021) - [Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning](https://mlsanity.com/paper/1708.00993/) (2017) - [ProTuner: Tuning Programs with Monte Carlo Tree Search](https://mlsanity.com/paper/2005.13685/) (2020) - [Moses: Efficient Exploitation of Cross-device Transferable Features for Tensor Program Optimization](https://mlsanity.com/paper/2201.05752/) (2022) ## Built on by (most cited first, 14 of 14) - [QiMeng-TensorOp: Automatically Generating High-Performance Tensor Operators with Hardware Primitives](https://mlsanity.com/paper/2505.06302/) (2025) - [CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs](https://mlsanity.com/paper/2311.09690/) (2023) - [PyGim: An Efficient Graph Neural Network Library for Real Processing-In-Memory Architectures](https://mlsanity.com/paper/2402.16731/) (2024) - [AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization](https://mlsanity.com/paper/2603.23566/) (2026) - [Pruner: A Draft-then-Verify Exploration Mechanism to Accelerate Tensor Program Tuning](https://mlsanity.com/paper/2402.02361/) (2024) - [GraphPerf-RT: A Graph-Driven Performance Model for Hardware-Aware Scheduling of OpenMP Codes](https://mlsanity.com/paper/2512.12091/) (2025) - [PassNet: Scaling Large Language Models for Graph Compiler Pass Generation](https://mlsanity.com/paper/2605.29357/) (2026) - [ML-driven Hardware Cost Model for MLIR](https://mlsanity.com/paper/2302.11405/) (2023) - [FTuner: A Fast Dynamic Shape Tensors Program Auto-Tuner for Deep Learning Compilers](https://mlsanity.com/paper/2407.21418/) (2024) - [Enhancing Deployment-Time Predictive Model Robustness for Code Analysis and Optimization](https://mlsanity.com/paper/2501.00298/) (2024) - [COGNATE: Acceleration of Sparse Tensor Programs on Emerging Hardware using Transfer Learning](https://mlsanity.com/paper/2506.00424/) (2025) - [Exploring the Feasibility of End-to-End Large Language Model as a Compiler](https://mlsanity.com/paper/2511.04132/) (2025) - [TCL: Enabling Fast and Efficient Cross-Hardware Tensor Program Optimization via Continual Learning](https://mlsanity.com/paper/2604.12891/) (2026) - [Hardware-Aware Features for CUTLASS Kernel Selection](https://mlsanity.com/paper/2609.35587/) (2026) ## Abstract (arXiv) Tensor program tuning is a non-convex objective optimization problem, to which search-based approaches have proven to be effective. At the core of the search-based approaches lies the design of the cost model. Though deep learning-based cost models perform significantly better than other methods, they still fall short and suffer from the following problems. First, their feature extraction heavily relies on expert-level domain knowledge in hardware architectures. Even so, the extracted features are often unsatisfactory and require separate considerations for CPUs and GPUs. Second, a cost model trained on one hardware platform usually performs poorly on another, a problem we call cross-hardware unavailability. In order to address these problems, we propose TLP and MTLTLP. TLP is a deep learning-based cost model that facilitates tensor program tuning. Instead of extracting features from the tensor program itself, TLP extracts features from the schedule primitives. We treat schedule primitives as tensor languages. TLP is thus a Tensor Language Processing task. In this way, the task of predicting the tensor program latency through the cost model is transformed into a natural language processing (NLP) regression task. MTL-TLP combines Multi-Task Learning and TLP to cope with the cross-hardware unavailability problem. We incorporate these techniques into the Ansor framework and conduct detailed experiments. Results show that TLP can speed up the average search time by 9.1X and 3.0X on CPU and GPU workloads, respectively, compared to the state-of-the-art implementation. MTL-TLP can achieve a speed-up of 4.7X and 2.9X on CPU and GPU workloads, respectively, using only 7% of the target hardware data.