mlsanityData to 14 Oct 2026

Reasoning-Intensive Regression

arXiv 2508.21762 ↗First posted 29 Aug 2025cs.CL cs.AIcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

AI researchers and practitioners increasingly apply large language models (LLMs) to what we call reasoning-intensive regression (RiR), i.e., deducing subtle numerical scores from text. Unlike standard language regression tasks such as sentiment or similarity analysis, RiR often appears instead in ad-hoc applications such as rubric-based scoring, modeling dense rewards in complex environments, or domain-specific retrieval, where much deeper analysis of context is required while only limited task-specific training data and computation are available. We cast four realistic problems as RiR tasks to establish an initial benchmark, and use that to test our hypothesis that prompting frozen LLMs and fine-tuning Transformer encoders via gradient descent will both often struggle in RiR. We then propose MENTAT, a simple and lightweight method that combines batch-reflective prompt optimization with neural ensemble learning. MENTAT achieves up to 65% improvement over both baselines, though substantial room remains for future advances.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 46 of its 80 references that are in the map. First posted on arXiv 29 Aug 2025.

Builds on

46 of 80 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025NeoBERT: A Next-Generation BERT26 Feb
  2. 2023Getting MoRE out of Mixture of Language Model Reasoning Experts24 May
  3. 2022Making Large Language Models Better Reasoners with Step-Aware Verifier6 Jun

Other references · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  7. 2017Decoupled Weight Decay Regularizationcited by 1,831
  8. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  9. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  10. 2025Kimi K2: Open Agentic Intelligencecited by 567
  11. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  12. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
31 more
  1. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 500
  2. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  3. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  4. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  5. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  6. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  7. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  8. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  9. 2024Generative Reward Modelscited by 148
  10. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  11. 2024Critique-out-Loud Reward Modelscited by 106
  12. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  13. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  14. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 57
  15. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  16. 2023The Expressive Power of Transformers with Chain of Thoughtcited by 50
  17. 2023Fusing Models with Complementary Expertisecited by 46
  18. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answeringcited by 43
  19. 2024Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoningcited by 40
  20. 2023Boosted Prompt Ensembles for Large Language Modelscited by 36
  21. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attentioncited by 34
  22. 2024OmniPred: Language Models as Universal Regressorscited by 34
  23. 2025Toward universal steering and monitoring of AI modelscited by 30
  24. 2024Understanding LLM Embeddings for Regressioncited by 19
  25. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judgescited by 14
  26. 2025Performance Prediction for Large Systems via Text-to-Text Regressioncited by 14
  27. 2025Decoding-based Regressioncited by 12
  28. 2024Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Modelscited by 11
  29. 2025WildIFEval: Instruction Following in the Wildcited by 11
  30. 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8
  31. 2024SpecFuse: Ensembling Large Language Models via Next-Segment Predictioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
  2. 2026Reinforcement Learning over Predictive Distributions for LLM Regressioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Large Language Models Are Not Strong Abstract Reasoners
  2. 2024Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks
  3. 2025Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Models
  4. 2026Task Complexity Matters: An Empirical Study of Reasoning in LLMs for Sentiment Analysis
  5. 2026A Survey of Reasoning-Intensive Retrieval: Progress and Challenges
  6. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders
  7. 2025Code-enabled language models can outperform reasoning models on diverse tasks
  8. 2025Training Language Models to Reason Efficiently