dIR -- Discrete Information Retrieval: Conversational Search over Unstructured (and Structured) Data with Large Language Models
Not on the map. Why?
Abstract (arXiv)
Data is stored in both structured and unstructured form. Querying both, to power natural language conversations, is a challenge. This paper introduces dIR, Discrete Information Retrieval, providing a unified interface to query both free text and structured knowledge. Specifically, a Large Language Model (LLM) transforms text into expressive representation. After the text is extracted into columnar form, it can then be queried via a text-to-SQL Semantic Parser, with an LLM converting natural language into SQL. Where desired, such conversation may be effected by a multi-step reasoning conversational agent. We validate our approach via a proprietary question/answer data set, concluding that dIR makes a whole new class of queries on free text possible when compared to traditionally fine-tuned dense-embedding-model-based Information Retrieval (IR) and SQL-based Knowledge Bases (KB). For sufficiently complex queries, dIR can succeed where no other method stands a chance.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 20 Dec 2023.
Builds on
30 of 42 references.
References · 30
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
- 2019Passage Re-ranking with BERTcited by 629
- 2023Enabling Large Language Models to Generate Text with Citationscited by 584
- 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
- 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 420
- 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
- 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
- 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 180
- 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
18 more
- 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 141
- 2017A Deep Relevance Matching Model for Ad-hoc Retrievalcited by 126
- 2019A Deep Look into Neural Ranking Models for Information Retrievalcited by 79
- 2023ReAcTable: Enhancing ReAct for Table Question Answeringcited by 76
- 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
- 2019Revealing the Importance of Semantic Retrieval for Machine Reading at Scalecited by 50
- 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
- 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 44
- 2020AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Datacited by 29
- 2023Text-to-SQL Error Correction with Language Models of Codecited by 21
- 2020DuoRAT: Towards Simpler Text-to-SQL Modelscited by 17
- 2019Content Enhanced BERT-based Text-to-SQL Generationcited by 16
- 2023Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrievalcited by 16
- 2023SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Datacited by 16
- 2019An Updated Duet Model for Passage Re-rankingcited by 12
- 2021Multi-Row, Multi-Span Distant Supervision For Table+Text Questioncited by 12
- 2023Adapt and Decompose: Efficient Generalization of Text-to-SQL via Domain Adapted Least-To-Most Promptingcited by 8
- 2020Revisiting the Open-Domain Question Answering Pipelinecited by 4
Built on by
Not cited yet.