mlsanityData to 7 Oct 2026

Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers

arXiv 2609.25237 ↗First posted 21 Sep 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned, evaluated, and deployed model under a budget, a human-approval gate, and reproducibility requirements. Seating an LLM agent in the FDE seat raises a question existing benchmarks cannot answer: not whether an agent can raise a metric, but whether it can be trusted to deliver. We answer it on a governed delivery plane, where an agent drives ten stages and an oracle scores each stage from platform-recorded facts. The central silent failure is the run that trains but does not learn (TBDL): loss falls, every signal stays green, and the delivered model is no better than the base. An operator-run acceptance gate catches every such run before payment, and a detector calibrated on known-corrupted runs flags severe corruption mid-run. We ran four frontier agents (Claude Opus 5, GPT-5.6-luna, Gemini 3.7 Flash, DeepSeek V4-Pro) end to end on metered L40S, A100, and H200 GPUs across 8B to 70B open bases, certifying every scenario before scoring. We also ran a human FDE arm under the same oracle and compare every agent against it.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 37 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

20 of 37 references.

References · 20

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  5. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  6. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  7. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  9. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  10. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  11. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 220
  12. 2020Estimating means of bounded random variables by bettingcited by 164
8 more
  1. 2022Game-theoretic statistics and safe anytime-valid inferencecited by 145
  2. 2026PostTrainBench: Can LLM Agents Automate LLM Post-Training?cited by 45
  3. 2025Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checkscited by 10
  4. 2025PRISM: Privacy-Aware Routing for Adaptive Cloud-Edge LLM Inference via Semantic Sketch Collaborationcited by 10
  5. 2026FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agentscited by 6
  6. 2026TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Explorationcited by 6
  7. 2026Orchestrating Data Collection and Computation in Green IoT Networkscited by 1
  8. 2026Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Interplay of Harness Design and Post-Training in LLM Agents
  2. 2026Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents
  3. 2026"LLM Agent Performance" Is Not a Single Evaluation Target
  4. 2026ττ-Bench: An Environment for End-To-End, Realistic Agent Construction
  5. 2026Adapting Technical-Service LLM Agents with Latent Logic Augmentation, Robust Noise Reduction, and Hybrid Reward Modeling
  6. 2026RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement
  7. 2026From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix
  8. 2026Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?