KnowledgeGain: Evaluating and Optimizing Science News Generation for Reader Learning
Not on the map. Why?
Abstract (arXiv)
Science news is an important medium to communicate discoveries between the research communities and the public. Yet, most metrics for generated or summarized text evaluate semantic similarity and factual consistency, but do not measure how much knowledge readers learn from the news. We introduce KnowledgeGain, a metric that evaluates the quality of science news by measuring how much knowledge readers gained after reading it. To evaluate the metric, we first performed a controlled human study and showed that the metric successfully captures the differential knowledge gained by human readers reading different types of science media. The data allowed us to calibrate a prompt-only LLM reader simulator. We use it to rank and filter candidate articles before human evaluation. A second human study shows that articles selected with this simulator improve post-reading accuracy and normalized KnowledgeGain over a strong generation baseline. Our work is a step toward generating science news that better meets the knowledge and comprehension goals of Bloom's Taxonomy.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 47 references that are in the map. First posted on arXiv 29 May 2026.
Builds on
16 of 47 references.
References · 16
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
- 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
- 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
- 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
- 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
- 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
- 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
- 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 105
- 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
- 2024Benchmarking Distributional Alignment of Large Language Modelscited by 61
4 more
- 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 57
- 2025Mixture-of-Personas Language Models for Population Simulationcited by 20
- 2020SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metricscited by 18
- 2023Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelitycited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.