Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ arrangement effect that is exactly zero under a convention-agnostic score
Not on the map. Why?
Abstract (arXiv)
"Train a model on the same problems written under two incompatible conventions, both correct, and ask what the ordering of that data writes into the parameters. The learning-rate schedule is not a background condition for that question. It is the averaging operator, and it decides the answer. We prove a bound in which the arrangement and the schedule enter the ordering effect as separate multiplied factors: the arrangement only as a block period, the schedule only as how much weight the endpoint can place on any one moment of the run. A decaying schedule cannot put a large step size and an uncontracted remainder at the same moment; a constant one does exactly that at the last step. That decay moderates ordering effects has been reported in pretraining; the mechanism, the separation, and a controlled measurement of both halves are ours. Ten orderings of one corpus, one budget, everything but the path held fixed, run twice under families differing in lr_scheduler_type and nothing else: at a constant rate the interior spans 0.2221 in allocation, 11.63 contrast floors, monotone in how blocked the arrangement is. Under the single cosine every published arm uses, the same ten arms occupy two distinguishable states where their own resolution would allow about ten. "Order matters" and "order does not matter" are the two ends of one knob. What the path writes is which convention the model commits to, and no exact-match benchmark can see it. Across twelve arms acc_A+acc_B is constant to within 9.7% while the allocation share runs 0.04 to 0.87, so the 12.29-sigma arrangement switch this paper measures is exactly zero under a convention-agnostic metric. That conservation is quoted from the decayed family throughout, the constant-rate one being a noisier place to read it. Marking the convention in the prompt collapses the switch and reaches 87.5% of the union ceiling."
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 82 references that are in the map. First posted on arXiv 23 Sep 2026.
Builds on
42 of 82 references.
References · 42
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
- 2024Knowledge Conflicts for LLMs: A Surveycited by 283
- 2024Continual Learning for Large Language Models: A Surveycited by 147
- 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 139
- 2018Don't forget, there is more than forgetting: new metrics for Continual Learningcited by 115
- 2020Random Reshuffling: Simple Analysis with Vast Improvementscited by 115
- 2023DoGE: Domain Reweighting with Generalization Estimationcited by 90
- 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 28
30 more
- 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 26
- 2025Metadata Conditioning Accelerates Language Model Pre-trainingcited by 22
- 2025Mid-Training of Large Language Models: A Surveycited by 21
- 2022GraB: Finding Provably Better Data Permutations than Random Reshufflingcited by 19
- 2025Data Mixing Can Induce Phase Transitions in Knowledge Acquisitioncited by 16
- 2024CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitionscited by 14
- 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
- 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 14
- 2025What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoningcited by 11
- 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 10
- 2026The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Ordercited by 9
- 2025When, Where and Why to Average Weights?cited by 8
- 2026Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Modelscited by 8
- 2025Influence Dynamics and Stagewise Data Attributioncited by 7
- 2026Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noisecited by 7
- 2025Fresh in memory: Training-order recency is linearly encoded in language model activationscited by 6
- 2026Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamicscited by 6
- 2026Learning is Forgetting: LLM Training As Lossy Compressioncited by 5
- 2024Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Mergingcited by 4
- 2026The Capability Convergence Hypothesis: Capability from Access Structure, Not Scalecited by 4
- 2026First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformerscited by 4
- 2025Interleaved Multitask Learning with Energy Modulated Learning Progresscited by 3
- 2025When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammarscited by 3
- 2026Demystifying Data Organization for Enhanced LLM Trainingcited by 3
- 2026The Order Is The Messagecited by 2
- 2026Learning to Shuffle: Block Reshuffling and Reversal Schemes for Stochastic Optimizationcited by 2
- 2026Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answercited by 2
- 2026Truth as a Compression Artifact in Language Model Trainingcited by 1
- 2026Shuffling the Data, Stretching the Step-size: Sharper Bias in constant step-size SGDcited by 1
- 2026Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEvalcited by 1
Built on by
Not cited yet.