# We Should Evaluate Real-World Impact

arXiv: https://arxiv.org/abs/2507.05973 · HTML: https://mlsanity.com/paper/2507.05973/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 8 Jul 2025.

## Builds on (most cited first, 11 of 11)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation](https://mlsanity.com/paper/2204.00447/) (2022)
- [PriMock57: A Dataset Of Primary Care Mock Consultations](https://mlsanity.com/paper/2204.00333/) (2022)
- [Ethics Sheets for AI Tasks](https://mlsanity.com/paper/2107.01183/) (2021)
- [Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projects](https://mlsanity.com/paper/2406.17910/) (2024)
- [Promises and pitfalls of artificial intelligence for legal applications](https://mlsanity.com/paper/2402.01656/) (2024)
- [DeepGen: Diverse Search Ad Generation and Real-Time Customization](https://mlsanity.com/paper/2208.03438/) (2022)
- [Knowledge Distillation based Contextual Relevance Matching for E-commerce Product Search](https://mlsanity.com/paper/2210.01701/) (2022)
- [Identifying High Consideration E-Commerce Search Queries](https://mlsanity.com/paper/2410.13951/) (2024)
- [Improving Retrieval in Sponsored Search by Leveraging Query Context Signals](https://mlsanity.com/paper/2407.14346/) (2024)
- [Contrastive Token Learning with Similarity Decay for Repetition Suppression in Machine Translation](https://mlsanity.com/paper/2409.19877/) (2024)

## Built on by (most cited first, 12 of 12)

- [DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence](https://mlsanity.com/paper/2509.04499/) (2025)
- [Interaction Protocol Shapes Moral Judgment in Multi-Agent Debate](https://mlsanity.com/paper/2510.10002/) (2025)
- [ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery](https://mlsanity.com/paper/2604.09237/) (2026)
- [The Environmental Impacts of Language Model Training Keep Rising Now is the Time to Catch Impacts on the Rebound](https://mlsanity.com/paper/2510.09022/) (2025)
- [NLG Evaluation: Past, Present, Future](https://mlsanity.com/paper/2605.23715/) (2026)
- [Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Tasks](https://mlsanity.com/paper/2602.12759/) (2026)
- [Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation](https://mlsanity.com/paper/2602.18483/) (2026)
- [Artificial intelligence language technologies in multilingual healthcare: Grand challenges ahead](https://mlsanity.com/paper/2605.01441/) (2026)
- [How Value Induction Reshapes LLM Behaviour](https://mlsanity.com/paper/2605.07925/) (2026)
- [A Good Talk Does not Look Like a Summary, It Teaches You! Measuring Takeaways from Paper-to-Video Talks](https://mlsanity.com/paper/2606.28531/) (2026)
- [Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries](https://mlsanity.com/paper/2607.16989/) (2026)
- [Beyond Good Intentions: When Does the Framing of Multilingual and Low-Resource NLP Research Become a Caricature?](https://mlsanity.com/paper/2608.30866/) (2026)

## Abstract (arXiv)

The ACL community has very little interest in evaluating the real-world impact of NLP systems. A structured survey of the ACL Anthology shows that perhaps 0.1% of its papers contain such evaluations; furthermore most papers which include impact evaluations present them very sketchily and instead focus on metric evaluations. NLP technology would be more useful and more quickly adopted if we seriously tried to understand and evaluate its real-world impact.