# Interpretability Framework for LLMs in Undergraduate Calculus

arXiv: https://arxiv.org/abs/2510.17910 · HTML: https://mlsanity.com/paper/2510.17910/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 19 Oct 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://mlsanity.com/paper/2402.01613/) (2024)
- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases](https://mlsanity.com/paper/2403.10446/) (2024)
- [Yo'LLaVA: Your Personalized Language and Vision Assistant](https://mlsanity.com/paper/2406.09400/) (2024)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Improving Natural Language Inference Using External Knowledge in the Science Questions Domain](https://mlsanity.com/paper/1809.05724/) (2018)
- [Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference](https://mlsanity.com/paper/2310.03184/) (2023)
- [Leveraging Large Language Models to Generate Answer Set Programs](https://mlsanity.com/paper/2307.07699/) (2023)
- [LLMs can Find Mathematical Reasoning Mistakes by Pedagogical Chain-of-Thought](https://mlsanity.com/paper/2405.06705/) (2024)
- [Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda](https://mlsanity.com/paper/2404.09554/) (2024)
- [FairAIED: Navigating Fairness, Bias, and Ethics in Educational AI Applications](https://mlsanity.com/paper/2407.18745/) (2024)
- [Learning Functional Causal Models with Generative Neural Networks](https://mlsanity.com/paper/1709.05321/) (2017)
- [Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Framework](https://mlsanity.com/paper/2501.15581/) (2025)
- [Context-Guided Dynamic Retrieval for Improving Generation Quality in RAG Models](https://mlsanity.com/paper/2504.19436/) (2025)
- [Large Language Models are Interpretable Learners](https://mlsanity.com/paper/2406.17224/) (2024)
- [Enhancing Health Information Retrieval with RAG by Prioritizing Topical Relevance and Factual Accuracy](https://mlsanity.com/paper/2502.04666/) (2025)
- [SBI-RAG: Enhancing Math Word Problem Solving for Students through Schema-Based Instruction and Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.13293/) (2024)
- [Human-Centric eXplainable AI in Education](https://mlsanity.com/paper/2410.19822/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly being used in education, yet their correctness alone does not capture the quality, reliability, or pedagogical validity of their problem-solving behavior, especially in mathematics, where multistep logic, symbolic reasoning, and conceptual clarity are critical. Conventional evaluation methods largely focus on final answer accuracy and overlook the reasoning process. To address this gap, we introduce a novel interpretability framework for analyzing LLM-generated solutions using undergraduate calculus problems as a representative domain. Our approach combines reasoning flow extraction and decomposing solutions into semantically labeled operations and concepts with prompt ablation analysis to assess input salience and output stability. Using structured metrics such as reasoning complexity, phrase sensitivity, and robustness, we evaluated the model behavior on real Calculus I to III university exams. Our findings revealed that LLMs often produce syntactically fluent yet conceptually flawed solutions, with reasoning patterns sensitive to prompt phrasing and input variation. This framework enables fine-grained diagnosis of reasoning failures, supports curriculum alignment, and informs the design of interpretable AI-assisted feedback tools. This is the first study to offer a structured, quantitative, and pedagogically grounded framework for interpreting LLM reasoning in mathematics education, laying the foundation for the transparent and responsible deployment of AI in STEM learning environments.