# Understanding LLM Embeddings for Regression

arXiv: https://arxiv.org/abs/2411.14708 · HTML: https://mlsanity.com/paper/2411.14708/

## Key facts

- Cited by 19 papers in the map (data through 7 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [SGD on Neural Networks Learns Functions of Increasing Complexity](https://mlsanity.com/paper/1905.11604/) (2019)
- [Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach](https://mlsanity.com/paper/1801.10578/) (2018)
- [Investigating the Limitations of Transformers with Simple Arithmetic Tasks](https://mlsanity.com/paper/2102.13019/) (2021)
- [From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples](https://mlsanity.com/paper/2404.07544/) (2024)
- [A Survey on Contextual Embeddings](https://mlsanity.com/paper/2003.07278/) (2020)
- [Benchmarking Neural Network Training Algorithms](https://mlsanity.com/paper/2306.07179/) (2023)
- [A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?](https://mlsanity.com/paper/2402.05015/) (2024)
- [OmniPred: Language Models as Universal Regressors](https://mlsanity.com/paper/2402.14547/) (2024)
- [Apollo: Transferable Architecture Exploration](https://mlsanity.com/paper/2102.01723/) (2021)
- [COCO: The Large Scale Black-Box Optimization Benchmarking (bbob-largescale) Test Suite](https://mlsanity.com/paper/1903.06396/) (2019)
- [Language Model Embeddings Can Be Sufficient for Bayesian Optimization](https://mlsanity.com/paper/2410.10190/) (2024)
- [Holistic Sentence Embeddings for Better Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07485/) (2022)

## Built on by (most cited first, 19 of 19)

- [Decoding-based Regression](https://mlsanity.com/paper/2501.19383/) (2025)
- [Hyperband-based Bayesian Optimization for Black-box Prompt Selection](https://mlsanity.com/paper/2412.07820/) (2024)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings](https://mlsanity.com/paper/2506.07109/) (2025)
- [ReViP: Mitigating False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance](https://mlsanity.com/paper/2601.16667/) (2026)
- [Reasoning-Intensive Regression](https://mlsanity.com/paper/2508.21762/) (2025)
- [Exploring the Synergy of Quantitative Factors and Newsflow Representations from Large Language Models for Stock Return Prediction](https://mlsanity.com/paper/2510.15691/) (2025)
- [MPM-LLM4DSE: Reaching the Pareto Frontier in HLS with Multimodal Learning and LLM-Driven Exploration](https://mlsanity.com/paper/2601.04801/) (2026)
- [Diffusion Large Language Models for Black-Box Optimization](https://mlsanity.com/paper/2601.14446/) (2026)
- [RELISH: LLM REgression with a Latent Iterative State Head](https://mlsanity.com/paper/2604.01206/) (2026)
- [Reinforcement Learning over Predictive Distributions for LLM Regression](https://mlsanity.com/paper/2605.20740/) (2026)
- [Semantic Embeddings of Chemical Elements for Enhanced Materials Inference and Discovery](https://mlsanity.com/paper/2502.14912/) (2025)
- [LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmark](https://mlsanity.com/paper/2505.12273/) (2025)
- [Transforming Decoder-Only Transformers for Accurate WiFi-Telemetry Based Indoor Localization](https://mlsanity.com/paper/2505.15835/) (2025)
- [Next Token Perception Score: Analytical Assessment of your LLM Perception Skills](https://mlsanity.com/paper/2505.17169/) (2025)
- [LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion](https://mlsanity.com/paper/2606.29900/) (2026)
- [From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation](https://mlsanity.com/paper/2607.12687/) (2026)
- [AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models](https://mlsanity.com/paper/2609.36368/) (2026)
- [Large Language Models are Approximate Survival Estimators](https://mlsanity.com/paper/2609.38181/) (2026)

## Abstract (arXiv)

With the rise of large language models (LLMs) for flexibly processing information as strings, a natural application is regression, specifically by preprocessing string representations into LLM embeddings as downstream features for metric prediction. In this paper, we provide one of the first comprehensive investigations into embedding-based regression and demonstrate that LLM embeddings as features can be better for high-dimensional regression tasks than using traditional feature engineering. This regression performance can be explained in part due to LLM embeddings over numeric data inherently preserving Lipschitz continuity over the feature space. Furthermore, we quantify the contribution of different model effects, most notably model size and language understanding, which we find surprisingly do not always improve regression performance.