# Knots: A Large-Scale Multi-Agent Enhanced Expert-Annotated Dataset and LLM Prompt Optimization for NOTAM Semantic Parsing

arXiv: https://arxiv.org/abs/2511.12630 · HTML: https://mlsanity.com/paper/2511.12630/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 16 Nov 2025.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [Chain-of-Verification Reduces Hallucination in Large Language Models](https://mlsanity.com/paper/2309.11495/) (2023)
- [Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View](https://mlsanity.com/paper/2310.02124/) (2023)
- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [Theory of Mind for Multi-Agent Collaboration via Large Language Models](https://mlsanity.com/paper/2310.10701/) (2023)
- [LLM Multi-Agent Systems: Challenges and Open Problems](https://mlsanity.com/paper/2402.03578/) (2024)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors](https://mlsanity.com/paper/2305.05711/) (2023)
- [Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2305.16582/) (2023)
- [Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models](https://mlsanity.com/paper/2407.06089/) (2024)
- [Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic](https://mlsanity.com/paper/2309.13339/) (2023)
- [KnowCoder: Coding Structured Knowledge into LLMs for Universal Information Extraction](https://mlsanity.com/paper/2403.07969/) (2024)
- [IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization](https://mlsanity.com/paper/2411.06208/) (2024)
- [Instance-adaptive Zero-shot Chain-of-Thought Prompting](https://mlsanity.com/paper/2409.20441/) (2024)
- [Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review](https://mlsanity.com/paper/2409.11864/) (2024)
- [Aviation Safety Enhancement via NLP & Deep Learning: Classifying Flight Phases in ATSB Safety Reports](https://mlsanity.com/paper/2501.07923/) (2025)
- [Integrating spoken instructions into flight trajectory prediction to optimize automation in air traffic control](https://mlsanity.com/paper/2305.01661/) (2023)

## Abstract (arXiv)

Notice to Air Missions (NOTAMs) serve as a critical channel for disseminating key flight safety information, yet their complex linguistic structures and implicit reasoning pose significant challenges for automated parsing. Existing research mainly focuses on surface-level tasks such as classification and named entity recognition, lacking deep semantic understanding. To address this gap, we propose NOTAM semantic parsing, a task emphasizing semantic inference and the integration of aviation domain knowledge to produce structured, inference-rich outputs. To support this task, we construct Knots (Knowledge and NOTAM Semantics), a high-quality dataset of 12,347 expert-annotated NOTAMs covering 194 Flight Information Regions, enhanced through a multi-agent collaborative framework for comprehensive field discovery. We systematically evaluate a wide range of prompt-engineering strategies and model-adaptation techniques, achieving substantial improvements in aviation text understanding and processing. Our experimental results demonstrate the effectiveness of the proposed approach and offer valuable insights for automated NOTAM analysis systems. Our code is available at: https://github.com/Estrellajer/Knots.