# SyllabusQA: A Course Logistics Question Answering Dataset

arXiv: https://arxiv.org/abs/2403.14666 · HTML: https://mlsanity.com/paper/2403.14666/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 3 Mar 2024.

## Builds on (most cited first, 14 of 14)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Fine-tuning Language Models for Factuality](https://mlsanity.com/paper/2311.08401/) (2023)
- [The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations](https://mlsanity.com/paper/1511.02301/) (2015)
- [Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension](https://mlsanity.com/paper/2203.13947/) (2022)
- [CliCR: A Dataset of Clinical Case Reports for Machine Reading Comprehension](https://mlsanity.com/paper/1803.09720/) (2018)
- [Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds](https://mlsanity.com/paper/1902.09093/) (2019)

## Built on by (most cited first, 9 of 9)

- [Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement](https://mlsanity.com/paper/2503.01875/) (2025)
- [Graphusion: A RAG Framework for Knowledge Graph Construction with a Global Perspective](https://mlsanity.com/paper/2410.17600/) (2024)
- [DeepRead: Document Structure-Aware Reasoning to Enhance Agentic Search](https://mlsanity.com/paper/2602.05014/) (2026)
- [LLM-as-a-qualitative-judge: automating error analysis in natural language generation](https://mlsanity.com/paper/2506.09147/) (2025)
- [EduVidQA: Generating and Evaluating Long-form Answers to Student Questions based on Lecture Videos](https://mlsanity.com/paper/2509.24120/) (2025)
- [Read the Docs Before Rewriting: Equip Rewriter with Domain Knowledge via Continual Pre-training](https://mlsanity.com/paper/2507.00477/) (2025)
- [PustakAI: Curriculum-Aligned and Interactive Textbooks Using Large Language Models](https://mlsanity.com/paper/2511.10002/) (2025)
- [Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos](https://mlsanity.com/paper/2609.01846/) (2026)
- [VikingRAG: Accurate and Token-efficient Retrieval-augmented Generation over Structured Documents](https://mlsanity.com/paper/2609.11390/) (2026)

## Abstract (arXiv)

Automated teaching assistants and chatbots have significant potential to reduce the workload of human instructors, especially for logistics-related question answering, which is important to students yet repetitive for instructors. However, due to privacy concerns, there is a lack of publicly available datasets. We introduce SyllabusQA, an open-source dataset with 63 real course syllabi covering 36 majors, containing 5,078 open-ended course logistics-related question-answer pairs that are diverse in both question types and answer formats. Since many logistics-related questions contain critical information like the date of an exam, it is important to evaluate the factuality of answers. We benchmark several strong baselines on this task, from large language model prompting to retrieval-augmented generation. We introduce Fact-QA, an LLM-based (GPT-4) evaluation metric to evaluate the factuality of predicted answers. We find that despite performing close to humans on traditional metrics of textual similarity, there remains a significant gap between automated approaches and humans in terms of fact precision.