mlsanityData to 14 Oct 2026

TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action

arXiv 2509.15098 ↗First posted 18 Sep 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Humanitarian Mine Action (HMA) addresses the challenge of detecting and removing landmines from conflict regions. Much of the life-saving operational knowledge produced by HMA agencies is buried in unstructured reports, limiting the transferability of information between agencies. To address this issue, we propose TextMineX: the first dataset, evaluation framework and ontology-guided large language model (LLM) pipeline for knowledge extraction from text in the HMA domain. TextMineX structures HMA reports into (subject, relation, object)-triples, thus creating domain-specific knowledge. To ensure real-world relevance, we utilized the dataset from our collaborator Cambodian Mine Action Centre (CMAC). We further introduce a bias-aware evaluation framework that combines human-annotated triples with an LLM-as-Judge protocol to mitigate position bias in reference-free scoring. Our experiments show that ontology-aligned prompts improve extraction accuracy by up to 44.2%, reduce hallucinations by 22.5%, and enhance format adherence by 20.9% compared to baseline models. We publicly release the dataset and code.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 84 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

42 of 84 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  8. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  9. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  10. 2023GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extractioncited by 119
  11. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 72
  12. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 69
30 more
  1. 2020Machine Knowledge: Creation and Curation of Comprehensive Knowledge Basescited by 55
  2. 2021GenIE: Generative Information Extractioncited by 53
  3. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  4. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  5. 2023Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Textcited by 48
  6. 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
  7. 2023Chainpoll: A high efficacy method for LLM hallucination detectioncited by 39
  8. 2024Evaluating Language Models as Synthetic Data Generatorscited by 33
  9. 2021Better than Average: Paired Evaluation of NLP Systemscited by 30
  10. 2024A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practicecited by 26
  11. 2019OPIEC: An Open Information Extraction Corpuscited by 20
  12. 2023Large Language Models for Biomedical Knowledge Graph Construction: Information extraction from EMR notescited by 18
  13. 2025Ontology-Guided Reverse Thinking Makes Large Language Models Stronger on Knowledge Graph Question Answeringcited by 15
  14. 2023PolyIE: A Dataset of Information Extraction from Polymer Material Scientific Literaturecited by 13
  15. 2024Enabling LLM Knowledge Analysis via Extensive Materializationcited by 12
  16. 2021BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluationcited by 11
  17. 2022Human-Centric Research for NLP: Towards a Definition and Guiding Questionscited by 9
  18. 2024Uncovering Limitations of Large Language Models in Information Seeking from Tablescited by 9
  19. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learningcited by 8
  20. 2021AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmarkcited by 7
  21. 2022Re-thinking Knowledge Graph Completion Evaluation from an Information Retrieval Perspectivecited by 7
  22. 2022HumSet: Dataset of Multilingual Information Extraction and Classification for Humanitarian Crisis Responsecited by 7
  23. 2018empathi: An ontology for Emergency Managing and Planning about Hazard Crisiscited by 6
  24. 2025Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMscited by 6
  25. 2023Hierarchical Evaluation Framework: Best Practices for Human Evaluationcited by 5
  26. 2023Linking Surface Facts to Large-Scale Knowledge Graphscited by 5
  27. 2022KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Modelscited by 4
  28. 2025FinKario: Event-Enhanced Automated Construction of Financial Knowledge Graphcited by 4
  29. 2024We Need to Talk About Classification Evaluation Metrics in NLPcited by 3
  30. 2021Developing Annotated Resources for Internal Displacement Monitoringcited by 2

Built on by

Not cited yet.