# Performance Prediction for Large Systems via Text-to-Text Regression

arXiv: https://arxiv.org/abs/2506.21718 · HTML: https://mlsanity.com/paper/2506.21718/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Builds on (most cited first, 11 of 11)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [Linear algebra with transformers](https://mlsanity.com/paper/2112.01898/) (2021)
- [From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples](https://mlsanity.com/paper/2404.07544/) (2024)
- [OmniPred: Language Models as Universal Regressors](https://mlsanity.com/paper/2402.14547/) (2024)
- [The Vizier Gaussian Process Bandit Algorithm](https://mlsanity.com/paper/2408.11527/) (2024)
- [Carbon accounting in the Cloud: a methodology for allocating emissions across data center users](https://mlsanity.com/paper/2406.09645/) (2024)

## Built on by (most cited first, 14 of 14)

- [Regression Language Models for Code](https://mlsanity.com/paper/2509.26476/) (2025)
- [LLM-Prior: A Framework for Knowledge-Driven Prior Elicitation and Aggregation](https://mlsanity.com/paper/2508.03766/) (2025)
- [Robustness is Important: Limitations of LLMs for Data Fitting](https://mlsanity.com/paper/2508.19563/) (2025)
- [Reasoning-Intensive Regression](https://mlsanity.com/paper/2508.21762/) (2025)
- [Estimating Time Series Foundation Model Transferability via In-Context Learning](https://mlsanity.com/paper/2509.23695/) (2025)
- [Configuration-to-Performance Scaling Law with Neural Ansatz](https://mlsanity.com/paper/2602.10300/) (2026)
- [LLMs can construct powerful representations and streamline sample-efficient supervised learning](https://mlsanity.com/paper/2603.11679/) (2026)
- [RELISH: LLM REgression with a Latent Iterative State Head](https://mlsanity.com/paper/2604.01206/) (2026)
- [AI for Distributed Systems Design: Scalable Cloud Optimization Through Repeated LLMs Sampling And Simulators](https://mlsanity.com/paper/2510.18897/) (2025)
- [Genomic Next-Token Predictors are In-Context Learners](https://mlsanity.com/paper/2511.12797/) (2025)
- [Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning](https://mlsanity.com/paper/2512.06533/) (2025)
- [LogSigma at SemEval-2026 Task 3: Uncertainty-Weighted Multitask Learning for Dimensional Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2603.24896/) (2026)
- [LASER: Language Model Regression for Semi-Structured Workflow Resource and Runtime Estimation](https://mlsanity.com/paper/2512.19701/) (2025)
- [Towards Reliable C-to-Rust Translation with Rule-Guided Reasoning and Reinforcement Learning](https://mlsanity.com/paper/2607.19966/) (2026)

## Abstract (arXiv)

In many industries, predicting metric outcomes of large systems is a fundamental problem, driven largely by traditional tabular regression. However, such methods struggle on complex systems data in the wild such as configuration files or system logs, where feature engineering is often infeasible. We propose text-to-text regression as a general, scalable alternative. For predicting resource efficiency on Borg, Google's massive compute cluster scheduling system, a 60M parameter encoder-decoder, trained from random initialization, achieves up to a near perfect 0.99 (0.9 average) rank correlation across the entire fleet, and 100x lower MSE than tabular approaches. The model also easily adapts to new tasks in only 500 few-shot examples and captures the densities of complex outcome distributions. Ablation studies highlight the importance of using encoders, increasing sequence length, and the model's inherent uncertainty quantification. These findings pave the way for universal simulators of real-world outcomes.