# Automate Knowledge Concept Tagging on Math Questions with LLMs

arXiv: https://arxiv.org/abs/2403.17281 · HTML: https://mlsanity.com/paper/2403.17281/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning](https://mlsanity.com/paper/2402.06332/) (2024)
- [EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction](https://mlsanity.com/paper/1906.05658/) (2019)
- [Structured information extraction from complex scientific text with fine-tuned large language models](https://mlsanity.com/paper/2212.05238/) (2022)
- [Active Learning Principles for In-Context Learning with Large Language Models](https://mlsanity.com/paper/2305.14264/) (2023)
- [Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection](https://mlsanity.com/paper/2310.20046/) (2023)
- [QuesNet: A Unified Representation for Heterogeneous Test Questions](https://mlsanity.com/paper/1905.10949/) (2019)
- [Instructed Language Models with Retrievers Are Powerful Entity Linkers](https://mlsanity.com/paper/2311.03250/) (2023)
- [DOCENT: Learning Self-Supervised Entity Representations from Large Document Collections](https://mlsanity.com/paper/2102.13247/) (2021)
- [More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineering](https://mlsanity.com/paper/2311.09782/) (2023)
- [Exploring the In-context Learning Ability of Large Language Model for Biomedical Concept Linking](https://mlsanity.com/paper/2307.01137/) (2023)
- [ChatEL: Entity Linking with Chatbots](https://mlsanity.com/paper/2402.14858/) (2024)
- [A Language Model based Framework for New Concept Placement in Ontologies](https://mlsanity.com/paper/2402.17897/) (2024)
- [Biomedical Entity Linking with Triple-aware Pre-Training](https://mlsanity.com/paper/2308.14429/) (2023)
- [Optimal strategies to perform multilingual analysis of social content for a novel dataset in the tourism domain](https://mlsanity.com/paper/2311.14727/) (2023)

## Built on by (most cited first, 6 of 6)

- [Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing](https://mlsanity.com/paper/2410.01727/) (2024)
- [Foundation Models for Education: Promises and Prospects](https://mlsanity.com/paper/2405.10959/) (2024)
- [CJEval: A Benchmark for Assessing Large Language Models Using Chinese Junior High School Exam Data](https://mlsanity.com/paper/2409.16202/) (2024)
- [Text-Based Approaches to Item Alignment to Content Standards in Large-Scale Reading & Writing Tests](https://mlsanity.com/paper/2509.26431/) (2025)
- [SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials](https://mlsanity.com/paper/2604.25472/) (2026)
- [Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning? A Perspective from Knowledge Space Theory](https://mlsanity.com/paper/2609.05245/) (2026)

## Abstract (arXiv)

Knowledge concept tagging for questions plays a crucial role in contemporary intelligent educational applications, including learning progress diagnosis, practice question recommendations, and course content organization. Traditionally, these annotations have been conducted manually with help from pedagogical experts, as the task requires not only a strong semantic understanding of both question stems and knowledge definitions but also deep insights into connecting question-solving logic with corresponding knowledge concepts. In this paper, we explore automating the tagging task using Large Language Models (LLMs), in response to the inability of prior manual methods to meet the rapidly growing demand for concept tagging in questions posed by advanced educational applications. Moreover, the zero/few-shot learning capability of LLMs makes them well-suited for application in educational scenarios, which often face challenges in collecting large-scale, expertise-annotated datasets. By conducting extensive experiments with a variety of representative LLMs, we demonstrate that LLMs are a promising tool for concept tagging in math questions. Furthermore, through case studies examining the results from different LLMs, we draw some empirical conclusions about the key factors for success in applying LLMs to the automatic concept tagging task.