mlsanityData to 7 Oct 2026

Evaluating Large Language Models with fmeval

arXiv 2407.12872 ↗First posted 15 Jul 2024cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

fmeval is an open source library to evaluate large language models (LLMs) in a range of tasks. It helps practitioners evaluate their model for task performance and along multiple responsible AI dimensions. This paper presents the library and exposes its underlying design principles: simplicity, coverage, extensibility and performance. We then present how these were implemented in the scientific and engineering choices taken when developing fmeval. A case study demonstrates a typical use case for the library: picking a suitable model for a question answering task. We close by discussing limitations and further work in the development of the library. fmeval can be found at https://github.com/aws/fmeval.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 25 of its 53 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

25 of 53 references.

References · 25

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2018Neural Network Acceptability Judgmentscited by 952
  11. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  12. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
13 more
  1. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  2. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  3. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  4. 2018Gender Bias in Coreference Resolutioncited by 438
  5. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  6. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  7. 2021Efficient Attentions for Long Document Summarizationcited by 342
  8. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  9. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  10. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  11. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  12. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  13. 2023Geographical Erasure in Language Generationcited by 15

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2025Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?cited by 49
  2. 2024ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMscited by 20
  3. 2025Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languagescited by 4