What Pretraining and Midtraining Make Learnable from Rewards?
Not on the map. Why?
Abstract (arXiv)
A reward can identify a correct answer while leaving the computation needed for new inputs undetermined. We study how pretraining and midtraining supply the information and computation that make reward adaptation effective. In sequential state computation and contextual memory, we characterize mechanisms that agree on every training reward yet demand different held-out answers. Task-independent source observations resolve this ambiguity. We construct finite sampled Adam paths from specified random initializations through source prediction and reward adaptation in the same parameters, proving how prediction acquires execution or retrieval and rewards learn their task-specific use. Experiments with pretrained Qwen2.5 checkpoints test this division of labor. Across eight worlds, Sequential models trained with correct source and first-operation supervision reach 82.61% success, versus 44.15% for a private-random source control. Memory replay preserves retrieval during reward adaptation, and an independent eight-world confirmation achieves 75.32% task success versus 49.86% after matched alternative-retrieval training. GSM8K and HotpotQA separate accuracy at reward entry, subsequent gain and final performance. Together, these results connect information acquisition, executable computation and reward-guided task learning.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 61 of its 86 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
61 of 86 references.
References · 61
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
- 2020On Layer Normalization in the Transformer Architecturecited by 922
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
- 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
38 more
- 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
- 2024Titans: Learning to Memorize at Test Timecited by 328
- 2025Defeating Prompt Injections by Designcited by 297
- 2022Transformers Learn Shortcuts to Automatacited by 268
- 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
- 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
- 2016DeepCoder: Learning to Write Programscited by 149
- 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
- 2024Transformers, parallel computation, and logarithmic depthcited by 79
- 2023Transformers as Support Vector Machinescited by 74
- 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 72
- 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 67
- 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 51
- 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 50
- 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 40
- 2023When can transformers reason with abstract symbols?cited by 24
- 2026daVinci-Dev: Agent-native Mid-training for Software Engineeringcited by 18
- 2024How Sparse Attention Approximates Exact Attention? Your Attention is Naturally nC-Sparsecited by 17
- 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 15
- 2026IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMscited by 15
- 2023Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weightscited by 12
- 2025How Reinforcement Learning After Next-Token Prediction Facilitates Learningcited by 10
- 2026On the Emergence of Implicit Curriculum in RLVR Learning Dynamicscited by 10
- 2024Learning and Transferring Sparse Contextual Bigrams with Linear Transformerscited by 9
- 2025Instruction Following by Principled Boosting Attention of Large Language Modelscited by 9
- 2025Provably Correct Automata Embeddings for Optimal Automata-Conditioned Reinforcement Learningcited by 6
- 2025When Can Model-Free Reinforcement Learning be Enough for Thinking?cited by 6
- 2026Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Datacited by 6
- 2026From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoningcited by 5
- 2024Logicbreaks: A Framework for Understanding Subversion of Rule-based Inferencecited by 4
- 2025On the Statistical Query Complexity of Learning Semiautomata: a Random Walk Approachcited by 4
- 2025Transformers Provably Learn Sparse XOR with Polylogarithmic Parameterscited by 3
- 2026Verify to Amplify: Improving Reasoning via Learned Chain-of-Thought Verificationcited by 3
- 2026Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledgecited by 3
- 2026A Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Trainingcited by 3
- 2026Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergencecited by 3
- 2026Learning to Reason with Curriculum II: Compositional Generalizationcited by 3
- 2026Verifiable Memory: Learning Unified Memory Management with Local and Global Verifiers for Large Language Model Agentscited by 3
50 most cited of 61 All 61 (JSON)
Built on by
Not cited yet.