mlsanityData to 7 Oct 2026

From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

arXiv 2404.07544 ↗First posted 11 Apr 2024cs.CL cs.AIcited by 55

Not on the map. Why?

Abstract (arXiv)

We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) can do linear and non-linear regression when given in-context examples, without any additional training or gradient updates. Our findings reveal that several large language models (e.g., GPT-4, Claude 3) are able to perform regression tasks with a performance rivaling (or even outperforming) that of traditional supervised methods such as Random Forest, Bagging, or Gradient Boosting. For example, on the challenging Friedman #2 regression dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM, Random Forest, KNN, or Gradient Boosting. We then investigate how well the performance of large language models scales with the number of in-context exemplars. We borrow from the notion of regret from online learning and empirically show that LLMs are capable of obtaining a sub-linear regret.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 40 of its 59 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

40 of 59 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Mistral 7Bcited by 3,065
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2024Mixtral of Expertscited by 1,785
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  11. 2022Transformers learn in-context by gradient descentcited by 714
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
28 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2023The Falcon Series of Open Language Modelscited by 526
  3. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 407
  4. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  5. 2023Larger language models do in-context learning differentlycited by 386
  6. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  7. 2023In-Context Learning Creates Task Vectorscited by 302
  8. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  9. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  10. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  11. 2023Large Language Models as General Pattern Machinescited by 196
  12. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  13. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  14. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  15. 2019Online Meta-Learningcited by 137
  16. 2023The Learnability of In-Context Learningcited by 126
  17. 2023In-Context Convergence of Transformerscited by 112
  18. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  19. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  20. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  21. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  22. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  23. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  24. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  25. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  26. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  27. 2023In-context Learning and Gradient Descent Revisitedcited by 20
  28. 2024Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Datacited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 29; 2026: 11.

55 papers.

Sort:

By area:

  1. 2024Many-Shot In-Context Learningcited by 203
  2. 2024How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensionscited by 51
  3. 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 37
  4. 2024InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generationcited by 30
  5. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  6. 2024Understanding LLM Embeddings for Regressioncited by 19
  7. 2025Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterizationcited by 15
  8. 2025Performance Prediction for Large Systems via Text-to-Text Regressioncited by 14
  9. 2025Decoding-based Regressioncited by 11
  10. 2024Fourier Head: Helping Large Language Models Learn Complex Probability Distributionscited by 10
  11. 2025Regression Language Models for Codecited by 10
  12. 2024Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictionscited by 9
38 more
  1. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  2. 202534 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific Discoverycited by 9
  3. 2025Quantile Regression with Large Language Models for Price Predictioncited by 9
  4. 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8
  5. 2024ELICIT: LLM Augmentation via External In-Context Capabilitycited by 7
  6. 2024Memorization in In-Context Learningcited by 6
  7. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  8. 2025Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoringcited by 6
  9. 2024A Survey of RWKVcited by 5
  10. 2026Eliciting Numerical Predictive Distributions of LLMs Without Autoregressioncited by 5
  11. 2024Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Modelscited by 4
  12. 2025ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Cipherscited by 4
  13. 2025LLM-Fusion: A Novel Multimodal Fusion Model for Accelerated Material Discoverycited by 3
  14. 2025Teaching Metric Distance to Discrete Autoregressive Language Modelscited by 3
  15. 2025On the Performance of LLMs for Real Estate Appraisalcited by 3
  16. 2025Towards Compute-Optimal Many-Shot In-Context Learningcited by 3
  17. 2025Improving the Distributional Alignment of LLMs using Supervisioncited by 2
  18. 2025Reasoning-Intensive Regressioncited by 2
  19. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  20. 2025Interpreting LLMs as Credit Risk Classifiers: Do Their Feature Explanations Align with Classical ML?cited by 2
  21. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  22. 2024Evaluating Theory of (an uncertain) Mind: Predicting the Uncertain Beliefs of Others in Conversation Forecastingcited by 1
  23. 2025Large language models streamline automated systematic review: A preliminary studycited by 1
  24. 2025Robustness is Important: Limitations of LLMs for Data Fittingcited by 1
  25. 2025PIE: Performance Interval Estimation for Free-Form Generation Taskscited by 1
  26. 2025Anticipatory Evaluation of Language Modelscited by 1
  27. 2025Towards Unified Vision Language Models for Forest Ecological Analysis in Earth Observationcited by 1
  28. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
  29. 2026LLM Flow Processes for Text-Conditioned Regressioncited by 1
  30. 2026RELISH: LLM REgression with a Latent Iterative State Headcited by 1
  31. 2026Reinforcement Learning over Predictive Distributions for LLM Regressioncited by 1
  32. 2024Scaling Particle Collision Data Analysiscited by 0
  33. 2025Transforming Decoder-Only Transformers for Accurate WiFi-Telemetry Based Indoor Localizationcited by 0
  34. 2025Next Token Perception Score: Analytical Assessment of your LLM Perception Skillscited by 0
  35. 2025Univariate to Multivariate: LLMs as Zero-Shot Predictors for Time-Series Forecastingcited by 0
  36. 2025In-Context Clustering with Large Language Modelscited by 0
  37. 2025LASER: Language Model Regression for Semi-Structured Workflow Resource and Runtime Estimationcited by 0
  38. 2026Federated Customization of Large Models: Approaches, Experiments, and Insightscited by 0

50 most cited of 55 All 55 (JSON)