# ISAAQ -- Mastering Textbook Questions with Pre-trained Transformers and Bottom-Up and Top-Down Attention

arXiv: https://arxiv.org/abs/2010.00562 · HTML: https://mlsanity.com/paper/2010.00562/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [Enhancing textual textbook question answering with large language models and retrieval augmented generation](https://mlsanity.com/paper/2402.05128/) (2024)
- [Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA](https://mlsanity.com/paper/2110.01552/) (2021)
- [Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trends](https://mlsanity.com/paper/2401.07518/) (2024)
- [Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI](https://mlsanity.com/paper/2510.04978/) (2025)
- [RL-CSDia: Representation Learning of Computer Science Diagrams](https://mlsanity.com/paper/2103.05900/) (2021)
- [Artificial Intelligence and Natural Language Processing and Understanding in Space: A Methodological Framework and Four ESA Case Studies](https://mlsanity.com/paper/2210.03640/) (2022)
- [Diagram-Driven Course Questions Generation](https://mlsanity.com/paper/2411.17771/) (2024)
- [Evaluating Multimodal Large Language Models on Educational Textbook Question Answering](https://mlsanity.com/paper/2506.21596/) (2025)
- [MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering](https://mlsanity.com/paper/2112.02839/) (2021)
- [Towards Language-driven Scientific AI](https://mlsanity.com/paper/2210.15327/) (2022)
- [Beyond Retrieval: Joint Supervision and Multimodal Document Ranking for Textbook Question Answering](https://mlsanity.com/paper/2505.13520/) (2025)
- [HyLoVQA: Dynamic Hypernetwork-Generated Low-Rank Adaptation for Continual Visual Question Answering](https://mlsanity.com/paper/2605.22035/) (2026)
- [Semantic Segmentation of Node and Edge Diagrams for Assistive Technology](https://mlsanity.com/paper/2606.11320/) (2026)
- [From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding](https://mlsanity.com/paper/2609.00948/) (2026)

## Abstract (arXiv)

Textbook Question Answering is a complex task in the intersection of Machine Comprehension and Visual Question Answering that requires reasoning with multimodal information from text and diagrams. For the first time, this paper taps on the potential of transformer language models and bottom-up and top-down attention to tackle the language and visual understanding challenges this task entails. Rather than training a language-visual transformer from scratch we rely on pre-trained transformers, fine-tuning and ensembling. We add bottom-up and top-down attention to identify regions of interest corresponding to diagram constituents and their relationships, improving the selection of relevant visual information for each question and answer options. Our system ISAAQ reports unprecedented success in all TQA question types, with accuracies of 81.36%, 71.11% and 55.12% on true/false, text-only and diagram multiple choice questions. ISAAQ also demonstrates its broad applicability, obtaining state-of-the-art results in other demanding datasets.