mlsanityData to 7 Oct 2026

We Should Evaluate Real-World Impact

arXiv 2507.05973 ↗First posted 8 Jul 2025cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

The ACL community has very little interest in evaluating the real-world impact of NLP systems. A structured survey of the ACL Anthology shows that perhaps 0.1% of its papers contain such evaluations; furthermore most papers which include impact evaluations present them very sketchily and instead focus on metric evaluations. NLP technology would be more useful and more quickly adopted if we seriously tried to understand and evaluate its real-world impact.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 11 of its 38 references that are in the map. First posted on arXiv 8 Jul 2025.

Builds on

11 of 38 references.

References · 11

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generationcited by 33
  3. 2022PriMock57: A Dataset Of Primary Care Mock Consultationscited by 30
  4. 2021Ethics Sheets for AI Taskscited by 18
  5. 2024Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projectscited by 18
  6. 2024Promises and pitfalls of artificial intelligence for legal applicationscited by 14
  7. 2022DeepGen: Diverse Search Ad Generation and Real-Time Customizationcited by 10
  8. 2022Knowledge Distillation based Contextual Relevance Matching for E-commerce Product Searchcited by 7
  9. 2024Identifying High Consideration E-Commerce Search Queriescited by 4
  10. 2024Improving Retrieval in Sponsored Search by Leveraging Query Context Signalscited by 2
  11. 2024Contrastive Token Learning with Similarity Decay for Repetition Suppression in Machine Translationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 9.

12 papers.

  1. 2025DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidencecited by 10
  2. 2026ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discoverycited by 3
  3. 2025The Environmental Impacts of Language Model Training Keep Rising Now is the Time to Catch Impacts on the Reboundcited by 2
  4. 2025Interaction Protocol Shapes Moral Judgment in Multi-Agent Debatecited by 2
  5. 2026NLG Evaluation: Past, Present, Futurecited by 2
  6. 2026Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Taskscited by 0
  7. 2026Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluationcited by 0
  8. 2026Artificial intelligence language technologies in multilingual healthcare: Grand challenges aheadcited by 0
  9. 2026How Value Induction Reshapes LLM Behaviourcited by 0
  10. 2026A Good Talk Does not Look Like a Summary, It Teaches You! Measuring Takeaways from Paper-to-Video Talkscited by 0
  11. 2026Real-World Evaluation of an AI Agent Drafting Translational Impact Summariescited by 0
  12. 2026Beyond Good Intentions: When Does the Framing of Multilingual and Low-Resource NLP Research Become a Caricature?cited by 0