# LLM Evaluation Based on Aerospace Manufacturing Expertise: Automated Generation and Multi-Model Question Answering

arXiv: https://arxiv.org/abs/2501.17183 · HTML: https://mlsanity.com/paper/2501.17183/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 24 references that are in the map.
- First posted on arXiv 25 Jan 2025.

## Builds on (most cited first, 4 of 4)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2404.05590/) (2024)
- [Quantifying Generalization Complexity for Large Language Models](https://mlsanity.com/paper/2410.01769/) (2024)

## Built on by (most cited first, 7 of 7)

- [SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data](https://mlsanity.com/paper/2505.20347/) (2025)
- [Rethinking the Generation of High-Quality CoT Data from the Perspective of LLM-Adaptive Question Difficulty Grading](https://mlsanity.com/paper/2504.11919/) (2025)
- [Machine Assistant with Reliable Knowledge: Enhancing Student Learning via RAG-based Retrieval](https://mlsanity.com/paper/2506.23026/) (2025)
- [CAMB: A comprehensive industrial LLM benchmark on civil aviation maintenance](https://mlsanity.com/paper/2508.20420/) (2025)
- [TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesis](https://mlsanity.com/paper/2604.08468/) (2026)
- [A Retrieval-Augmented Language Assistant for Unmanned Aircraft Safety Assessment and Regulatory Compliance](https://mlsanity.com/paper/2603.09999/) (2026)
- [Downsides of Smartness Across Edge-Cloud Continuum in Modern Industry](https://mlsanity.com/paper/2603.29289/) (2026)

## Abstract (arXiv)

Aerospace manufacturing demands exceptionally high precision in technical parameters. The remarkable performance of Large Language Models (LLMs), such as GPT-4 and QWen, in Natural Language Processing has sparked industry interest in their application to tasks including process design, material selection, and tool information retrieval. However, LLMs are prone to generating "hallucinations" in specialized domains, producing inaccurate or false information that poses significant risks to the quality of aerospace products and flight safety. This paper introduces a set of evaluation metrics tailored for LLMs in aerospace manufacturing, aiming to assess their accuracy by analyzing their performance in answering questions grounded in professional knowledge. Firstly, key information is extracted through in-depth textual analysis of classic aerospace manufacturing textbooks and guidelines. Subsequently, utilizing LLM generation techniques, we meticulously construct multiple-choice questions with multiple correct answers of varying difficulty. Following this, different LLM models are employed to answer these questions, and their accuracy is recorded. Experimental results demonstrate that the capabilities of LLMs in aerospace professional knowledge are in urgent need of improvement. This study provides a theoretical foundation and practical guidance for the application of LLMs in aerospace manufacturing, addressing a critical gap in the field.