# Interpretation of Natural Language Rules in Conversational Machine Reading

arXiv: https://arxiv.org/abs/1809.01494 · HTML: https://mlsanity.com/paper/1809.01494/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2018.

## Built on by (most cited first, 30 of 75)

- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [ChatQA: Surpassing GPT-4 on Conversational QA and RAG](https://mlsanity.com/paper/2401.10225/) (2024)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Dialog Inpainting: Turning Documents into Dialogs](https://mlsanity.com/paper/2205.09073/) (2022)
- [MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents](https://mlsanity.com/paper/2109.12595/) (2021)
- [MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2501.03468/) (2025)
- [The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions](https://mlsanity.com/paper/2310.12418/) (2023)
- [TWEETSUMM -- A Dialog Summarization Dataset for Customer Service](https://mlsanity.com/paper/2111.11894/) (2021)
- [Conversational Question Answering: A Survey](https://mlsanity.com/paper/2106.00874/) (2021)
- [ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers](https://mlsanity.com/paper/2110.06884/) (2021)
- [HybriDialogue: An Information-Seeking Dialogue Dataset Grounded on Tabular and Textual Data](https://mlsanity.com/paper/2204.13243/) (2022)
- [INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactions](https://mlsanity.com/paper/2207.00746/) (2022)
- [Varifocal Question Generation for Fact-checking](https://mlsanity.com/paper/2210.12400/) (2022)
- [English Machine Reading Comprehension Datasets: A Survey](https://mlsanity.com/paper/2101.10421/) (2021)
- [Asking Clarification Questions to Handle Ambiguity in Open-Domain QA](https://mlsanity.com/paper/2305.13808/) (2023)
- [ISEEQ: Information Seeking Question Generation using Dynamic Meta-Information Retrieval and Knowledge Graphs](https://mlsanity.com/paper/2112.07622/) (2021)
- [DIALKI: Knowledge Identification in Conversational Systems through Dialogue-Document Contextualization](https://mlsanity.com/paper/2109.04673/) (2021)
- [Factoring Statutory Reasoning as Language Understanding Challenges](https://mlsanity.com/paper/2105.07903/) (2021)
- [Database Reasoning Over Text](https://mlsanity.com/paper/2106.01074/) (2021)
- [End-to-End Learning of Flowchart Grounded Task-Oriented Dialogs](https://mlsanity.com/paper/2109.07263/) (2021)
- [Ditch the Gold Standard: Re-evaluating Conversational Question Answering](https://mlsanity.com/paper/2112.08812/) (2021)
- [Chain of Logic: Rule-Based Reasoning with Large Language Models](https://mlsanity.com/paper/2402.10400/) (2024)
- [Generating Information-Seeking Conversations from Unlabeled Documents](https://mlsanity.com/paper/2205.12609/) (2022)
- [When and What to Ask: AskBench and Rubric-Guided RLVR for LLM Clarification](https://mlsanity.com/paper/2602.11199/) (2026)
- [Advances in Multi-turn Dialogue Comprehension: A Survey](https://mlsanity.com/paper/2103.03125/) (2021)
- [Open-Retrieval Conversational Machine Reading](https://mlsanity.com/paper/2102.08633/) (2021)
- [DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queries](https://mlsanity.com/paper/2310.04678/) (2023)
- [Iterative Hierarchical Attention for Answering Complex Questions over Long Documents](https://mlsanity.com/paper/2106.00200/) (2021)
- [DG2: Data Augmentation Through Document Grounded Dialogue Generation](https://mlsanity.com/paper/2112.08342/) (2021)

## Abstract (arXiv)

Most work in machine reading focuses on question answering problems where the answer is directly expressed in the text to read. However, many real-world question answering problems require the reading of text not because it contains the literal answer, but because it contains a recipe to derive an answer together with the reader's background knowledge. One example is the task of interpreting regulations to answer "Can I...?" or "Do I have to...?" questions such as "I am working in Canada. Do I have to carry on paying UK National Insurance?" after reading a UK government website about this topic. This task requires both the interpretation of rules and the application of background knowledge. It is further complicated due to the fact that, in practice, most questions are underspecified, and a human assistant will regularly have to ask clarification questions such as "How long have you been working abroad?" when the answer cannot be directly derived from the question and text. In this paper, we formalise this task and develop a crowd-sourcing strategy to collect 32k task instances based on real-world rules and crowd-generated questions and scenarios. We analyse the challenges of this task and assess its difficulty by evaluating the performance of rule-based and machine-learning baselines. We observe promising results when no background knowledge is necessary, and substantial room for improvement whenever background knowledge is needed.