mlsanityData to 7 Oct 2026

Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes

arXiv 2304.09433 ↗First posted 19 Apr 2023cs.CLcited by 101

Not on the map. Why?

Abstract (arXiv)

A long standing goal of the data management community is to develop general, automated systems that ingest semi-structured documents and output queryable tables without human effort or domain specific customization. Given the sheer variety of potential documents, state-of-the art systems make simplifying assumptions and use domain specific training. In this work, we ask whether we can maintain generality by using large language models (LLMs). LLMs, which are pretrained on broad data, can perform diverse downstream tasks simply conditioned on natural language task descriptions. We propose and evaluate EVAPORATE, a simple, prototype system powered by LLMs. We identify two fundamentally different strategies for implementing this system: prompt the LLM to directly extract values from documents or prompt the LLM to synthesize code that performs the extraction. Our evaluations show a cost-quality tradeoff between these two approaches. Code synthesis is cheap, but far less accurate than directly processing each document with the LLM. To improve quality while maintaining low cost, we propose an extended code synthesis implementation, EVAPORATE-CODE+, which achieves better quality than direct extraction. Our key insight is to generate many candidate functions and ensemble their extractions using weak supervision. EVAPORATE-CODE+ not only outperforms the state-of-the art systems, but does so using a sublinear pass over the documents with the LLM. This equates to a 110x reduction in the number of tokens the LLM needs to process, averaged across 16 real-world evaluation settings of 10k documents each.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 31 of its 80 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

31 of 80 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  11. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  12. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
19 more
  1. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  2. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  3. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  4. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  5. 2022Can Foundation Models Wrangle Your Data?cited by 173
  6. 2022A Survey on Retrieval-Augmented Text Generationcited by 157
  7. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  8. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
  9. 2020Interactive Weak Supervision: Learning Useful Heuristics for Data Labelingcited by 52
  10. 2018A Survey on Open Information Extractioncited by 51
  11. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 51
  12. 2018Neural Open Information Extractioncited by 50
  13. 2019Learning Dependency Structures for Weak Supervision Modelscited by 32
  14. 2020ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpagescited by 30
  15. 2022A Survey on Neural Open Information Extraction: Current Status and Future Directionscited by 27
  16. 2022Operationalizing Machine Learning: An Interview Studycited by 25
  17. 2022DOM-LM: Learning Generalizable Representations for HTML Documentscited by 24
  18. 2022Reasoning over Public and Private Data in Retrieval-Based Systemscited by 21
  19. 2015Incremental Knowledge Base Construction Using DeepDivecited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 18; 2025: 35; 2026: 37.

101 papers.

Sort:

By area:

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  2. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  4. 2023Large Language Models as Tool Makerscited by 293
  5. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  6. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  7. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  8. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  9. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  10. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 54
  11. 2024DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processingcited by 49
  12. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
38 more
  1. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  2. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  3. 2024A Declarative System for Optimizing AI Workloadscited by 35
  4. 2024Reward-Robust RLHF in LLMscited by 32
  5. 2024Smoothie: Label Free Language Model Routingcited by 31
  6. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 30
  7. 2024Semantic Operators: A Declarative Model for Rich, AI-based Data Processingcited by 29
  8. 2023CHORUS: Foundation Models for Unified Data Discovery and Explorationcited by 24
  9. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 24
  10. 2023Jellyfish: A Large Language Model for Data Preprocessingcited by 20
  11. 2024The Design of an LLM-powered Unstructured Analytics Systemcited by 20
  12. 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Modelscited by 17
  13. 2023Embedding-based Retrieval with LLM for Effective Agriculture Information Extracting from Unstructured Datacited by 16
  14. 2025BookRAG: A Hierarchical Structure-aware Index-based Approach for Retrieval-Augmented Generation on Complex Documentscited by 16
  15. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  16. 2025A Practical Guide for Designing, Developing, and Deploying Production-Grade Agentic AI Workflowscited by 13
  17. 2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
  18. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  19. 2025Agentsway -- Software Development Methodology for AI Agents-based Teamscited by 12
  20. 2025Towards Responsible and Explainable AI Agents with Consensus-Driven Reasoningcited by 12
  21. 2026Memory Caching: RNNs with Growing Memorycited by 12
  22. 2023From Words to Code: Harnessing Data for Program Synthesis from Natural Languagecited by 11
  23. 2025ST-Raptor: LLM-Powered Semi-Structured Table Question Answeringcited by 11
  24. 2025Standardization of Psychiatric Diagnoses -- Role of Fine-tuned LLM Consortium and OpenAI-gpt-oss Reasoning LLM Enabled Decision Support Systemcited by 11
  25. 2025Cortex AISQL: A Production SQL Engine for Unstructured Datacited by 11
  26. 2023SEED: Domain-Specific Data Curation With Large Language Modelscited by 10
  27. 2024E2ETune: End-to-End Knob Tuning via Fine-tuned Generative Language Modelcited by 9
  28. 2026Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMscited by 9
  29. 2025Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guaranteescited by 8
  30. 2025A Survey on Retrieval And Structuring Augmented Generation with Large Language Modelscited by 8
  31. 2024When Reasoning Meets Information Aggregation: A Case Study with Sports Narrativescited by 7
  32. 2024Mind the Data Gap: Bridging LLMs to Enterprise Data Integrationcited by 7
  33. 2025AGRAG: Advanced Graph-based Retrieval-Augmented Generation for LLMscited by 7
  34. 2025Visual Template Inference for Data Extraction from Documentscited by 6
  35. 2024Chameleon: Foundation Models for Fairness-aware Multi-modal Data Augmentation to Enhance Coverage of Minoritiescited by 5
  36. 2024CARROT: A Learned Cost-Constrained Retrieval Optimization System for RAGcited by 5
  37. 2026Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrencescited by 5
  38. 2025SQUiD: Synthesizing Relational Databases from Unstructured Textcited by 4

50 most cited of 101 All 101 (JSON)