# SIREN: Situation-aware Index for Response and Emergency ActioNability in Crisis Translation

arXiv: https://arxiv.org/abs/2610.16646 · HTML: https://mlsanity.com/paper/2610.16646/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 30 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Builds on (most cited first, 7 of 7)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [TICO-19: the Translation Initiative for Covid-19](https://mlsanity.com/paper/2007.01788/) (2020)
- [HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation](https://mlsanity.com/paper/2112.13833/) (2021)
- [The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control](https://mlsanity.com/paper/2405.16969/) (2024)
- [Is MT Ready for the Next Crisis or Pandemic?](https://mlsanity.com/paper/2601.10082/) (2026)
- [LLM-Powered Automatic Translation and Urgency in Crisis Scenarios](https://mlsanity.com/paper/2602.13452/) (2026)
- [Translating Under Pressure: Domain-Aware LLMs for Crisis Communication](https://mlsanity.com/paper/2604.26597/) (2026)

## Abstract (arXiv)

Machine translation (MT) is increasingly utilized in multilingual crisis communication, yet conventional evaluation does not directly assess whether translated emergency messages preserve their actionability. We introduce SIREN (Situation-aware Index for Response and Emergency ActioNability), a human evaluation framework assessing whether translation choices compromise protective action, directive force, and urgency of the source text. We evaluate SIREN in two complementary settings. At the error level, we apply it to HOPE-annotated Italian--English translations produced by ChatGPT-4o and Google Translate: 31.63% and 48.73% of errors, respectively, compromise actionability, predominantly by affecting the protective action. At the message level, we evaluate English--French translations produced by TranslateGemma-4B and Google Translate, with actionability preserved in 85.34% and 92.67% of translations, respectively. TranslateGemma-4B failures predominantly weaken directive force, whereas Google Translate failures more often compromise protective actions and receive Major impact ratings. These findings show that translation systems can differ not only in how often actionability is lost, but also in the nature and functional impact of these failures. Alongside SIREN, we contribute an English--French crisis-translation resource with human actionability annotations and post-edited reference translations, supporting further research on function-oriented evaluation for crisis communication. Our experimental data is available at https://github.com/mcstaiano/SIREN.