LLaMa-SciQ: An Educational Chatbot for Answering Science MCQ
On the map: LLM reasoning → Mathematical reasoning
Abstract (arXiv)
Large Language Models (LLMs) often struggle with tasks requiring mathematical reasoning, particularly multiple-choice questions (MCQs). To address this issue, we developed LLaMa-SciQ, an educational chatbot designed to assist college students in solving and understanding MCQs in STEM fields. We begin by fine-tuning and aligning the models to human preferences. After comparing the performance of Mistral-7B and LLaMa-8B, we selected the latter as the base model due to its higher evaluation accuracy. To further enhance accuracy, we implement Retrieval-Augmented Generation (RAG) and apply quantization to compress the model, reducing inference time and increasing accessibility for students. For mathematical reasoning, LLaMa-SciQ achieved 74.5% accuracy on the GSM8k dataset and 30% on the MATH dataset. However, RAG does not improve performance and even reduces it, likely due to retriever issues or the model's unfamiliarity with context. Despite this, the quantized model shows only a 5% loss in performance, demonstrating significant efficiency improvements.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 37 references that are in the map. First posted on arXiv 25 Sep 2024.
Builds on
20 of 37 references.
Closest ancestors · 1
Same direction, newest first
Other references · 19
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
7 more
- 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
- 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
- 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
- 2024Direct Preference Optimization with an Offsetcited by 122
- 2021Including Signed Languages in Natural Language Processingcited by 67
- 2023Better Zero-Shot Reasoning with Self-Adaptive Promptingcited by 52
- 2023Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Codecited by 31
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
- 2025Towards Efficient Educational Chatbots: Benchmarking RAG Frameworks
- 2024Comparing the Utility, Preference, and Performance of Course Material Search Functionality and Retrieval-Augmented Generation Large Language Model (RAG-LLM) AI Chatbots in Information-Seeking Tasks
- 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agents
- 2024Enhancing Critical Thinking in Education by means of a Socratic Chatbot
- 2024MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions
- 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering
- 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systems