mlsanityData to 7 Oct 2026

RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement

arXiv 2605.09730 ↗First posted 10 May 2026cs.LG cs.SEcited by 4

Not on the map. Why?

Abstract (arXiv)

Iterative self-refinement is a popular inference-time reliability technique, but its effectiveness in code-mode tool use depends heavily on the structure of the feedback signal: unstructured critique helps inconsistently across models, and even revision with real execution feedback improves only modestly. The dominant failures are inter-tool contract violations (wrong output shape, incorrect tool routing, broken argument provenance) that run to completion without raising errors, making runtime feedback insufficient. We introduce RubricRefine, a training-free method for pre-execution contract checking that generates task- and registry-specific rubrics, scores candidate code against explicit contract checks, and iteratively repairs failures before any execution occurs. RubricRefine reaches 0.86, averaged across seven models, on M3ToolEval with zero execution attempts, improving over prior inference-time baselines at lower latency than rubric-guided reranking. Performance remains flat on the predominantly single-step API-Bank, consistent with the method's reliance on inter-tool contract structure. Results on AppWorld demonstrate that our method maintains an advantage in the multi-turn setting. Because the rubric is derived from the supplied tool documentation, the method's advantage survives incomplete documentation but reverses under incorrect documentation. A rubric-category ablation identifies which rules are load-bearing, and top-bin calibration enables early stopping even where aggregate calibration is poor.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 226 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

43 of 226 references.

References · 43

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  12. 2023Teaching Large Language Models to Self-Debugcited by 963
31 more
  1. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  2. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  3. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  4. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  5. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  6. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  7. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  8. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  9. 2022CodeT: Code Generation with Generated Testscited by 369
  10. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  11. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  12. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  13. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  14. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  15. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  16. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  17. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 100
  18. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 80
  19. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 67
  20. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 58
  21. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
  22. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 40
  23. 2025Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learningcited by 22
  24. 2026Agentic Rubrics as Contextual Verifiers for SWE Agentscited by 22
  25. 2023Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Modelscited by 17
  26. 2023GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolutioncited by 17
  27. 2025Graph RAG-Tool Fusioncited by 15
  28. 2019Accurate Layerwise Interpretable Competence Estimationcited by 10
  29. 2025Chain-of-Tools: Utilizing Massive Unseen Tools in the CoT Reasoning of Frozen Language Modelscited by 9
  30. 2026PreFlect: From Retrospective to Prospective Reflection in Large Language Model Agentscited by 4
  31. 2025Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria Rerankingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 25
  2. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  3. 2026Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agentscited by 0
  4. 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0