# Leveraging pre-trained language models for conversational information seeking from text

arXiv: https://arxiv.org/abs/2204.03542 · HTML: https://mlsanity.com/paper/2204.03542/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization](https://mlsanity.com/paper/2110.07356/) (2021)
- [Detecting Hate Speech with GPT-3](https://mlsanity.com/paper/2103.12407/) (2021)
- [ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning](https://mlsanity.com/paper/2005.00771/) (2020)
- [A Few More Examples May Be Worth Billions of Parameters](https://mlsanity.com/paper/2110.04374/) (2021)
- [An Approach for Process Model Extraction By Multi-Grained Text Classification](https://mlsanity.com/paper/1906.02127/) (2019)

## Built on by (most cited first, 3 of 3)

- [PAGED: A Benchmark for Procedural Graphs Extraction from Documents](https://mlsanity.com/paper/2408.03630/) (2024)
- [Procedural Text Mining with Large Language Models](https://mlsanity.com/paper/2310.03376/) (2023)
- [Multi-Agent Procedural Graph Extraction with Structural and Logical Refinement](https://mlsanity.com/paper/2601.19170/) (2026)

## Abstract (arXiv)

Recent advances in Natural Language Processing, and in particular on the construction of very large pre-trained language representation models, is opening up new perspectives on the construction of conversational information seeking (CIS) systems. In this paper we investigate the usage of in-context learning and pre-trained language representation models to address the problem of information extraction from process description documents, in an incremental question and answering oriented fashion. In particular we investigate the usage of the native GPT-3 (Generative Pre-trained Transformer 3) model, together with two in-context learning customizations that inject conceptual definitions and a limited number of samples in a few shot-learning fashion. The results highlight the potential of the approach and the usefulness of the in-context learning customizations, which can substantially contribute to address the "training data challenge" of deep learning based NLP techniques the BPM field. It also highlight the challenge posed by control flow relations for which further training needs to be devised.