mlsanityData to 14 Oct 2026

Decoding-based Regression

arXiv 2501.19383 ↗First posted 31 Jan 2025cs.LG cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Language models have recently been shown capable of performing regression wherein numeric predictions are represented as decoded strings. In this work, we provide theoretical grounds for this capability and furthermore investigate the utility of causal sequence decoding models as numeric regression heads given any feature representation. We find that, despite being trained in the usual way - for next-token prediction via cross-entropy loss - decoder-based heads are as performant as standard pointwise heads when benchmarked over standard regression tasks, while being flexible enough to capture smooth numeric distributions, such as in the task of density estimation.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 25 of its 48 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

25 of 48 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2018Hierarchical Neural Story Generationcited by 1,193
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  8. 2014OpenML: networked science in machine learningcited by 526
  9. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  10. 2024Generative Reward Modelscited by 148
  11. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  12. 2019Rank consistent ordinal regression for neural networks with application to age estimationcited by 133
13 more
  1. 2020Bayesian Neural Networks: An Introduction and Surveycited by 131
  2. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  3. 2018Improving Regression Performance with Distributional Lossescited by 67
  4. 2021Linear algebra with transformerscited by 63
  5. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 57
  6. 2022AMLB: an AutoML Benchmarkcited by 55
  7. 2022Deep Symbolic Regression for Recurrent Sequencescited by 54
  8. 2018A no-regret generalization of hierarchical softmax to extreme multi-label classificationcited by 52
  9. 2024OmniPred: Language Models as Universal Regressorscited by 34
  10. 2024Understanding LLM Embeddings for Regressioncited by 19
  11. 2024Regression-aware Inference with LLMscited by 14
  12. 2019COCO: The Large Scale Black-Box Optimization Benchmarking (bbob-largescale) Test Suitecited by 12
  13. 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

  1. 2025Regression Language Models for Codecited by 11
  2. 2026The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?cited by 10
  3. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  4. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 7
  5. 2025Meta-Statistical Learning: Supervised Learning of Statistical Estimatorscited by 3
  6. 2025Reasoning-Intensive Regressioncited by 2
  7. 2026RELISH: LLM REgression with a Latent Iterative State Headcited by 2
  8. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
  9. 2025LASER: Language Model Regression for Semi-Structured Workflow Resource and Runtime Estimationcited by 0
  10. 2026FlowTool: Controlling Tool Parameter in Image Retouching via Flow Matchingcited by 0
  11. 2026Large Language Models are Approximate Survival Estimatorscited by 0
  12. 2026Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stoppingcited by 0