mlsanityData to 14 Oct 2026

MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferences

arXiv 2410.02381 ↗First posted 3 Oct 2024cs.CL cs.AI cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

Understanding the quality of a performance evaluation metric is crucial for ensuring that model outputs align with human preferences. However, it remains unclear how well each metric captures the diverse aspects of these preferences, as metrics often excel in one particular area but not across all dimensions. To address this, it is essential to systematically calibrate metrics to specific aspects of human preference, catering to the unique characteristics of each aspect. We introduce MetaMetrics, a calibrated meta-metric designed to evaluate generation tasks across different modalities in a supervised manner. MetaMetrics optimizes the combination of existing metrics to enhance their alignment with human preferences. Our metric demonstrates flexibility and effectiveness in both language and vision downstream tasks, showing significant benefits across various multilingual and multi-domain scenarios. MetaMetrics aligns closely with human preferences and is highly extendable and easily integrable into any application. This makes MetaMetrics a powerful tool for improving the evaluation of generation tasks, ensuring that metrics are more representative of human judgment across diverse contexts.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 55 of its 84 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

55 of 84 references.

References · 55

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
38 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  9. 2015Teaching Machines to Read and Comprehendcited by 1,049
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  12. 2018A Tutorial on Bayesian Optimizationcited by 744
  13. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  14. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  15. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  16. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  17. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  18. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  19. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  20. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  21. 2024InternLM2 Technical Reportcited by 397
  22. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  23. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  24. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  25. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  26. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  27. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 184
  28. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  29. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 146
  30. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  31. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  32. 2021A survey on high-dimensional Gaussian process modeling with application to Bayesian optimizationcited by 77
  33. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  34. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  35. 2019Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentencescited by 55
  36. 2021Lower Perplexity is Not Always Human-Likecited by 52
  37. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
  38. 2021Transparent Human Evaluation for Image Captioningcited by 40

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  2. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  3. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 23
  4. 2025Reviving The Classics: Active Reward Modeling in Large Language Model Alignmentcited by 8
  5. 2024Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Modelscited by 6
  6. 2024MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibrationcited by 6
  7. 2025Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountabilitycited by 6
  8. 2025SAFER: Probing Safety in Reward Models with Sparse Autoencodercited by 6
  9. 2024CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedbackcited by 5
  10. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUscited by 4
  11. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  12. 2026Calibrating Model-Based Evaluation Metrics for Summarizationcited by 3
3 more
  1. 2025Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluationcited by 1
  2. 2025Tiny Reward Modelscited by 0
  3. 2026Dynamic Meta-Metrics: Source-Sentence Conditioned Weighting for MT Evaluationcited by 0