mlsanityData to 7 Oct 2026

LMUnit: Fine-grained Evaluation with Natural Language Unit Tests

arXiv 2412.13091 ↗First posted 17 Dec 2024cs.CL cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

As language models become integral to critical workflows, assessing their behavior remains a fundamental challenge -- human evaluation is costly and noisy, while automated metrics provide only coarse, difficult-to-interpret signals. We introduce natural language unit tests, a paradigm that decomposes response quality into explicit, testable criteria, along with a unified scoring model, LMUnit, which combines multi-objective training across preferences, direct ratings, and natural language rationales. Through controlled human studies, we show this paradigm significantly improves inter-annotator agreement and enables more effective LLM development workflows. LMUnit achieves state-of-the-art performance on evaluation benchmarks (FLASK, BigGenBench) and competitive results on RewardBench. These results validate both our proposed paradigm and scoring model, suggesting a promising path forward for language model evaluation and development.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 56 of its 86 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

56 of 86 references.

References · 56

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
38 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  2. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  4. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  5. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  6. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  7. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  9. 2022CodeT: Code Generation with Generated Testscited by 369
  10. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  11. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  12. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  13. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  14. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  15. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  16. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  17. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  18. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  19. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  20. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  21. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  22. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  23. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  24. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  25. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  26. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  27. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  28. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  29. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  30. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  31. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
  32. 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
  33. 2023Human Feedback is not Gold Standardcited by 82
  34. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  35. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 67
  36. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  37. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 59
  38. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 55

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 13.

32 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  3. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  4. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  5. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  6. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  7. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  8. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  9. 2025TD-EVAL: Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisonscited by 10
  10. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 8
  11. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewardscited by 7
  12. 2025Bridging Human and LLM Judgments: Understanding and Narrowing the Gapcited by 6
20 more
  1. 2025Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Valuecited by 6
  2. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  3. 2025Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluationcited by 3
  4. 2025Evalet: Evaluating Large Language Models through Functional Fragmentationcited by 3
  5. 2025On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalizationcited by 3
  6. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
  7. 2025AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agentscited by 2
  8. 2026Entropy Aware Reward Guidance for Diffusion Language Model Alignmentcited by 2
  9. 2026FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Modelscited by 2
  10. 2026Assessing the Pedagogical Readiness of Large Language Models as AI Tutors in Low-Resource Contexts: A Case Study of Nepal's K-10 Curriculumcited by 1
  11. 2026Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubricscited by 1
  12. 2026Grounded Checklist Partial Credit for Agent Skill Trajectoriescited by 1
  13. 2025MutaGReP: Execution-Free Repository-Grounded Plan Search for Code-Usecited by 0
  14. 2025The AI Consumer Index (ACE)cited by 0
  15. 2026Stable Reasoning, Unstable Responses: Mitigating LLM Deception via Stability Asymmetrycited by 0
  16. 2026From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMscited by 0
  17. 2026Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregationcited by 0
  18. 2026Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judgescited by 0
  19. 2026CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluationcited by 0
  20. 2026FastGuide: Accelerating Reward Guidance for Diffusion Large Language Modelscited by 0