mlsanityData to 7 Oct 2026

FAER: Auditable Utility-Aligned Trajectory Replay for Language Model Post-Training

arXiv 2610.00385 ↗First posted 30 Sep 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Replay selectors often rank cached trajectories by format feedback, confidence, freshness, or response length, although cache-level correctness and downstream learner utility are distinct objectives. We formalize this selection-to-learning gap and introduce FAER as an auditable full-trajectory replay framework. Its training-free fixed selector is a protocol baseline; FAER-UTILITY is the learner-aware selector fitted on disjoint calibration blocks. The normalized gradient alignment is reported as a baseline, while a disposable optimizer-aware virtual update supplies a magnitude-aware utility surface. The audit contract freezes observed fields and replay traces before evaluation labels are joined. On GSM8K with Qwen2.5-1.5B-Instruct, the matched learner study reports quality 0.6329 for the fixed selector, compared with 0.5482 for uniform and 0.6037 for format-feedback under 128 updates. Metadata-only cross-fitted calibration reaches 0.6476!±!0.0139 over eight seeds (median 0.6481; paired 95% interval [+0.079,+0.122]) at 63,276 target-run tokens; its recorded full cost is 189,642 tokens and 3.48 GPU-hours including calibration. The completed FAER-UTILITY row reaches 0.6624 at 62,844 target-run tokens and 4.26 GPU-hours. Format-feedback selects records with correctness 0.6953, compared with 0.3594 for the fixed selector, despite the different downstream ranking. The completed comparison surfaces report the learner-aware ablation, same-seed gap, policy-optimization rows, and strict zero-shot transfer.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

31 of 39 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  11. 2023Let's Verify Step by Stepcited by 3,598
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
19 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2018Datasheets for Datasetscited by 1,371
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2018Model Cards for Model Reportingcited by 1,068
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  13. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  14. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  15. 2017Deep Reinforcement Learning that Matterscited by 727
  16. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 318
  17. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  18. 2020Revisiting Fundamentals of Experience Replaycited by 171
  19. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164

Built on by

Not cited yet.