# NoTeS-Bank: Benchmarking Neural Transcription and Search for Scientific Notes Understanding

arXiv: https://arxiv.org/abs/2504.09249 · HTML: https://mlsanity.com/paper/2504.09249/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 54 references that are in the map.
- First posted on arXiv 12 Apr 2025.

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://mlsanity.com/paper/2105.14517/) (2021)
- [MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations](https://mlsanity.com/paper/2407.01523/) (2024)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://mlsanity.com/paper/2007.15207/) (2020)
- [DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis](https://mlsanity.com/paper/2206.01062/) (2022)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [Hierarchical multimodal transformers for Multi-Page DocVQA](https://mlsanity.com/paper/2212.05935/) (2022)
- [ScreenAI: A Vision-Language Model for UI and Infographics Understanding](https://mlsanity.com/paper/2402.04615/) (2024)
- [TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domains](https://mlsanity.com/paper/2404.19205/) (2024)
- [JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation](https://mlsanity.com/paper/1910.02216/) (2019)
- [Document Collection Visual Question Answering](https://mlsanity.com/paper/2104.14336/) (2021)
- [CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checking](https://mlsanity.com/paper/2206.11863/) (2022)
- [CodeQA: A Question Answering Dataset for Source Code Comprehension](https://mlsanity.com/paper/2109.08365/) (2021)
- [A Question Answering Approach to Emotion Cause Extraction](https://mlsanity.com/paper/1708.05482/) (2017)
- [Document Visual Question Answering Challenge 2020](https://mlsanity.com/paper/2008.08899/) (2020)
- [MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding](https://mlsanity.com/paper/2410.21311/) (2024)
- [NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset](https://mlsanity.com/paper/2109.10604/) (2021)

## Built on by (most cited first, 3 of 3)

- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Benchmarking Vision-Language Models for French PDF-to-Markdown Conversion](https://mlsanity.com/paper/2602.11960/) (2026)
- [WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans](https://mlsanity.com/paper/2608.22959/) (2026)

## Abstract (arXiv)

Understanding and reasoning over academic handwritten notes remains a challenge in document AI, particularly for mathematical equations, diagrams, and scientific notations. Existing visual question answering (VQA) benchmarks focus on printed or structured handwritten text, limiting generalization to real-world note-taking. To address this, we introduce NoTeS-Bank, an evaluation benchmark for Neural Transcription and Search in note-based question answering. NoTeS-Bank comprises complex notes across multiple domains, requiring models to process unstructured and multimodal content. The benchmark defines two tasks: (1) Evidence-Based VQA, where models retrieve localized answers with bounding-box evidence, and (2) Open-Domain VQA, where models classify the domain before retrieving relevant documents and answers. Unlike classical Document VQA datasets relying on optical character recognition (OCR) and structured data, NoTeS-BANK demands vision-language fusion, retrieval, and multimodal reasoning. We benchmark state-of-the-art Vision-Language Models (VLMs) and retrieval frameworks, exposing structured transcription and reasoning limitations. NoTeS-Bank provides a rigorous evaluation with NDCG@5, MRR, Recall@K, IoU, and ANLS, establishing a new standard for visual document understanding and reasoning.