# A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation

arXiv: https://arxiv.org/abs/2004.07633 · HTML: https://mlsanity.com/paper/2004.07633/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2020.

## Built on by (most cited first, 5 of 5)

- [ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems](https://mlsanity.com/paper/2306.04743/) (2023)
- [Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries](https://mlsanity.com/paper/2402.08349/) (2024)
- [StatBot.Swiss: Bilingual Open Data Exploration in Natural Language](https://mlsanity.com/paper/2406.03170/) (2024)
- [INODE: Building an End-to-End Data Exploration System in Practice [Extended Vision]](https://mlsanity.com/paper/2104.04194/) (2021)
- [DeKeyNLU: Enhancing Natural Language to SQL Generation through Task Decomposition and Keyword Extraction](https://mlsanity.com/paper/2509.14507/) (2025)

## Abstract (arXiv)

In this paper, we introduce a novel methodology to efficiently construct a corpus for question answering over structured data. For this, we introduce an intermediate representation that is based on the logical query plan in a database called Operation Trees (OT). This representation allows us to invert the annotation process without losing flexibility in the types of queries that we generate. Furthermore, it allows for fine-grained alignment of query tokens to OT operations. In our method, we randomly generate OTs from a context-free grammar. Afterwards, annotators have to write the appropriate natural language question that is represented by the OT. Finally, the annotators assign the tokens to the OT operations. We apply the method to create a new corpus OTTA (Operation Trees and Token Assignment), a large semantic parsing corpus for evaluating natural language interfaces to databases. We compare OTTA to Spider and LC-QuaD 2.0 and show that our methodology more than triples the annotation speed while maintaining the complexity of the queries. Finally, we train a state-of-the-art semantic parsing model on our data and show that our corpus is a challenging dataset and that the token alignment can be leveraged to increase the performance significantly.