mlsanityData to 14 Oct 2026

LLMs in the Real World: Evaluating "AI" in Emergency Contexts

arXiv 2607.00019 ↗First posted 29 May 2026cs.CY cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper offers a call to action. We urge our colleagues in the research community to play a greater role in the articulation of our findings to the public. To illustrate the stakes we present a case study on the initial stages of an LLM-based machine translation application's deployment in a real-world context: a text-2-911 system advertising capabilities in 55 languages for use in emergencies in which it may be difficult to call operators directly. We identify a number of common misconceptions about technologies such as these, concluding with a set of concrete recommendations and best practices for stakeholders at every stage of the development and deployment pipeline. While the advancement of scientific research often lies in solving the "hard" problems, we argue it is often the "easy" ones -- problems for which the latest technology is often unnecessary -- that are most overlooked.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 80 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

20 of 80 references.

References · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2018Model Cards for Model Reportingcited by 1,080
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  4. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  5. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  6. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 71
  7. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  8. 2023The Elephant in the Room: Analyzing the Presence of Big Tech in Natural Language Processing Researchcited by 27
  9. 2024SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakescited by 24
  10. 2024Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?cited by 23
  11. 2023Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errorscited by 22
  12. 2024Shortcomings of LLMs for Low-Resource Translation: Retrieval and Understanding are Both the Problemcited by 17
8 more
  1. 2025Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoningcited by 9
  2. 2019Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT qualitycited by 7
  3. 2024Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillationcited by 6
  4. 2025Leveraging Domain Knowledge at Inference Time for LLM Translation: Retrieval versus Generationcited by 4
  5. 2025You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translationcited by 4
  6. 2025Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translationcited by 4
  7. 2024No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Sizecited by 3
  8. 2024Ethical Concern Identification in NLP: A Corpus of ACL Anthology Ethics Statementscited by 3

Built on by

Not cited yet.