mlsanityData to 7 Oct 2026

Understanding LLM Embeddings for Regression

arXiv 2411.14708 ↗First posted 22 Nov 2024cs.LG cs.AI cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

With the rise of large language models (LLMs) for flexibly processing information as strings, a natural application is regression, specifically by preprocessing string representations into LLM embeddings as downstream features for metric prediction. In this paper, we provide one of the first comprehensive investigations into embedding-based regression and demonstrate that LLM embeddings as features can be better for high-dimensional regression tasks than using traditional feature engineering. This regression performance can be explained in part due to LLM embeddings over numeric data inherently preserving Lipschitz continuity over the feature space. Furthermore, we quantify the contribution of different model effects, most notably model size and language understanding, which we find surprisingly do not always improve regression performance.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

21 of 30 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  9. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  10. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  11. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  12. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
9 more
  1. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  2. 2020A Survey on Contextual Embeddingscited by 48
  3. 2023Benchmarking Neural Network Training Algorithmscited by 46
  4. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
  5. 2024OmniPred: Language Models as Universal Regressorscited by 34
  6. 2021Apollo: Transferable Architecture Explorationcited by 17
  7. 2019COCO: The Large Scale Black-Box Optimization Benchmarking (bbob-largescale) Test Suitecited by 12
  8. 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8
  9. 2022Holistic Sentence Embeddings for Better Out-of-Distribution Detectioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025Decoding-based Regressioncited by 11
  2. 2024Hyperband-based Bayesian Optimization for Black-box Prompt Selectioncited by 9
  3. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  4. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 6
  5. 2026ReViP: Mitigating False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalancecited by 5
  6. 2025Reasoning-Intensive Regressioncited by 2
  7. 2025Exploring the Synergy of Quantitative Factors and Newsflow Representations from Large Language Models for Stock Return Predictioncited by 1
  8. 2026MPM-LLM4DSE: Reaching the Pareto Frontier in HLS with Multimodal Learning and LLM-Driven Explorationcited by 1
  9. 2026Diffusion Large Language Models for Black-Box Optimizationcited by 1
  10. 2026RELISH: LLM REgression with a Latent Iterative State Headcited by 1
  11. 2026Reinforcement Learning over Predictive Distributions for LLM Regressioncited by 1
  12. 2025Semantic Embeddings of Chemical Elements for Enhanced Materials Inference and Discoverycited by 0
7 more
  1. 2025LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmarkcited by 0
  2. 2025Transforming Decoder-Only Transformers for Accurate WiFi-Telemetry Based Indoor Localizationcited by 0
  3. 2025Next Token Perception Score: Analytical Assessment of your LLM Perception Skillscited by 0
  4. 2026LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusioncited by 0
  5. 2026From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimationcited by 0
  6. 2026AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Modelscited by 0
  7. 2026Large Language Models are Approximate Survival Estimatorscited by 0