# Towards Efficient Educational Chatbots: Benchmarking RAG Frameworks

arXiv: https://arxiv.org/abs/2503.00781 · HTML: https://mlsanity.com/paper/2503.00781/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation.
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 2 Mar 2025.

## Placed in

- Retrieval-augmented generation / [Agentic RAG, agents and memory](https://mlsanity.com/rag/agentic-rag-agents-and-memory/)

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering](https://mlsanity.com/paper/2210.02627/) (2022)
- [Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation](https://mlsanity.com/paper/2404.05970/) (2024)
- [Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey](https://mlsanity.com/paper/2311.07914/) (2023)
- [Evaluating Retrieval Quality in Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.13781/) (2024)
- [A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions](https://mlsanity.com/paper/2410.12837/) (2024)
- [Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximization](https://mlsanity.com/paper/2405.02816/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) have proven immensely beneficial in education by capturing vast amounts of literature-based information, allowing them to generate context without relying on external sources. In this paper, we propose a generative AI-powered GATE question-answering framework (GATE stands for Graduate Aptitude Test in Engineering) that leverages LLMs to explain GATE solutions and support students in their exam preparation. We conducted extensive benchmarking to select the optimal embedding model and LLM, evaluating our framework based on criteria such as latency, faithfulness, and relevance, with additional validation through human evaluation. Our chatbot integrates state-of-the-art embedding models and LLMs to deliver accurate, context-aware responses. Through rigorous experimentation, we identified configurations that balance performance and computational efficiency, ensuring a reliable chatbot to serve students' needs. Additionally, we discuss the challenges faced in data processing and modeling and implemented solutions. Our work explores the application of Retrieval-Augmented Generation (RAG) for GATE Q/A explanation tasks, and our findings demonstrate significant improvements in retrieval accuracy and response quality. This research offers practical insights for developing effective AI-driven educational tools while highlighting areas for future enhancement in usability and scalability.