mlsanityData to 7 Oct 2026

OmniPred: Language Models as Universal Regressors

arXiv 2402.14547 ↗First posted 22 Feb 2024cs.LG cs.AI cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

Regression is a powerful tool to accurately predict the outcome metric of a system given a set of parameters, but has traditionally been restricted to methods which are only applicable to a specific task. In this paper, we propose OmniPred, a framework for training language models as universal end-to-end regressors over (x,y) data from arbitrary formats. Using data sourced from Google Vizier, one of the largest proprietary blackbox optimization databases in the world, our extensive experiments demonstrate that language models are capable of very precise numerical regression using only textual representations of mathematical parameters and values, and if given the opportunity to train at scale over multiple tasks, can significantly outperform traditional regression models.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 33 of its 52 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

33 of 52 references.

References · 33

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  12. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
21 more
  1. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  2. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  3. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 398
  4. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  5. 2020Neural Architecture Search without Trainingcited by 241
  6. 2020Training independent subnetworks for robust predictioncited by 177
  7. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  8. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 96
  9. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  10. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 70
  11. 2020Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarkscited by 67
  12. 2021Linear algebra with transformerscited by 62
  13. 2022Deep Symbolic Regression for Recurrent Sequencescited by 53
  14. 2018Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networkscited by 43
  15. 2021Data-Driven Offline Optimization For Architecting Hardware Acceleratorscited by 29
  16. 2018Learning Memory Access Patternscited by 28
  17. 2022Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimizationcited by 17
  18. 2019COCO: The Large Scale Black-Box Optimization Benchmarking (bbob-largescale) Test Suitecited by 12
  19. 2023Transfer Learning for Bayesian Optimization on Heterogeneous Search Spacescited by 9
  20. 2020Neural Architecture Performance Prediction Using Graph Neural Networkscited by 8
  21. 2023Unlocking the Transferability of Tokens in Deep Models for Tabular Datacited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 19; 2026: 6.

34 papers.

Sort:

By area:

  1. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  2. 2024Understanding LLM Embeddings for Regressioncited by 19
  3. 2024Reinforced In-Context Black-Box Optimizationcited by 14
  4. 2025Offline Model-Based Optimization: Comprehensive Reviewcited by 14
  5. 2025Performance Prediction for Large Systems via Text-to-Text Regressioncited by 14
  6. 2025Decoding-based Regressioncited by 11
  7. 2024Fourier Head: Helping Large Language Models Learn Complex Probability Distributionscited by 10
  8. 2025Regression Language Models for Codecited by 10
  9. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  10. 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8
  11. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 6
  12. 2024RiskLabs: Predicting Financial Risk Using Large Language Model based on Multimodal and Multi-Sources Datacited by 4
22 more
  1. 2025Teaching Metric Distance to Discrete Autoregressive Language Modelscited by 3
  2. 2024What explains the success of cross-modal fine-tuning with ORCA?cited by 2
  3. 2025Hybrid Reasoning for Perception, Explanation, and Autonomous Action in Manufacturingcited by 2
  4. 2025Improving the Distributional Alignment of LLMs using Supervisioncited by 2
  5. 2025Reasoning-Intensive Regressioncited by 2
  6. 2025FoMEMO: Towards Foundation Models for Expensive Multi-objective Optimizationcited by 2
  7. 2025Interpreting LLMs as Credit Risk Classifiers: Do Their Feature Explanations Align with Classical ML?cited by 2
  8. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  9. 2026LLMs can construct powerful representations and streamline sample-efficient supervised learningcited by 2
  10. 2025Anticipatory Evaluation of Language Modelscited by 1
  11. 2025ZeroShotOpt: Towards Zero-Shot Pretrained Models for Efficient Black-Box Optimizationcited by 1
  12. 2025Exploring the Synergy of Quantitative Factors and Newsflow Representations from Large Language Models for Stock Return Predictioncited by 1
  13. 2025Towards Unified Vision Language Models for Forest Ecological Analysis in Earth Observationcited by 1
  14. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
  15. 2026RELISH: LLM REgression with a Latent Iterative State Headcited by 1
  16. 2026Support-Proximity Augmented Diffusion Estimation for Offline Black-Box Optimizationcited by 1
  17. 2026Reinforcement Learning over Predictive Distributions for LLM Regressioncited by 1
  18. 2024SysCaps: Language Interfaces for Simulation Surrogates of Complex Systemscited by 0
  19. 2024Sample-Efficient Bayesian Optimization with Transfer Learning for Heterogeneous Search Spacescited by 0
  20. 2025ALF: Advertiser Large Foundation Model for Multi-Modal Advertiser Understandingcited by 0
  21. 2025LASER: Language Model Regression for Semi-Structured Workflow Resource and Runtime Estimationcited by 0
  22. 2026The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Eracited by 0