mlsanityData to 14 Oct 2026

Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration

arXiv 2601.07224 ↗First posted 12 Jan 2026cs.AI cs.LGcited by 1

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

While Hybrid Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become the standard paradigm for training LLM agents, effective mechanisms for data allocation between these stages remain largely underexplored. Current data arbitration strategies often rely on surface-level heuristics that fail to diagnose intrinsic learning needs. Since SFT targets pattern consolidation through imitation while RL drives structural adaptation via exploration, misaligning data with these functional roles causes severe optimization interference. We propose PRISM, a dynamics-aware framework grounded in Schema Theory that arbitrates data based on its degree of cognitive conflict with the model's existing knowledge. By analyzing the spatial geometric structure of gradients, PRISM identifies data triggering high spatial concentration as high-conflict signals that require RL for structural restructuring. In contrast, data yielding diffuse updates is routed to SFT for efficient consolidation. Extensive experiments on WebShop and ALFWorld demonstrate that PRISM achieves a Pareto improvement, outperforming state-of-the-art hybrid methods while reducing computational costs by up to 3.22×. Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 30 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

20 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  2. 2025ToolRL: Reward is All Tool Learning Needs16 Apr

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  10. 2023LIMA: Less Is More for Alignmentcited by 1,100
  11. 2018On Lazy Training in Differentiable Programmingcited by 675
  12. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
6 more
  1. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  2. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  3. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  4. 2020Estimating Example Difficulty Using Variance of Gradientscited by 97
  5. 2024Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearningcited by 15
  6. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
  2. 2026Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting
  3. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
  4. 2026Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysis
  5. 2026Behavior Knowledge Merge in Reinforced Agentic Models
  6. 2026RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents
  7. 2026Spectral Rewiring for Exploration, Purification, and Model Merging
  8. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning