# Instruct and Extract: Instruction Tuning for On-Demand Information Extraction

arXiv: https://arxiv.org/abs/2310.16040 · HTML: https://mlsanity.com/paper/2310.16040/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness](https://mlsanity.com/paper/2304.11633/) (2023)
- [MAVEN: A Massive General Domain Event Detection Dataset](https://mlsanity.com/paper/2004.13590/) (2020)
- [Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detection](https://mlsanity.com/paper/1910.11621/) (2019)
- [Joint Entity and Relation Extraction with Set Prediction Networks](https://mlsanity.com/paper/2011.01675/) (2020)
- [Text-to-Table: A New Way of Information Extraction](https://mlsanity.com/paper/2109.02707/) (2021)
- [Table-to-Text: Describing Table Region with Natural Language](https://mlsanity.com/paper/1805.11234/) (2018)
- [Low-Resource Name Tagging Learned with Weakly Labeled Data](https://mlsanity.com/paper/1908.09659/) (2019)
- [Open-Vocabulary Argument Role Prediction for Event Extraction](https://mlsanity.com/paper/2211.01577/) (2022)

## Built on by (most cited first, 30 of 36)

- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature](https://mlsanity.com/paper/2406.07835/) (2024)
- [ADELIE: Aligning Large Language Models on Information Extraction](https://mlsanity.com/paper/2405.05008/) (2024)
- [Instruction-Tuning LLMs for Event Extraction with Annotation Guidelines](https://mlsanity.com/paper/2502.16377/) (2025)
- [A Simple but Effective Approach to Improve Structured Language Model Output for Information Extraction](https://mlsanity.com/paper/2402.13364/) (2024)
- [A Survey on Retrieval And Structuring Augmented Generation with Large Language Models](https://mlsanity.com/paper/2509.10697/) (2025)
- [Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation](https://mlsanity.com/paper/2410.16597/) (2024)
- [IEPile: Unearthing Large-Scale Schema-Based Information Extraction Corpus](https://mlsanity.com/paper/2402.14710/) (2024)
- [Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Datasets](https://mlsanity.com/paper/2406.05348/) (2024)
- [Investigating Instruction Tuning Large Language Models on Graphs](https://mlsanity.com/paper/2408.05457/) (2024)
- [BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuning](https://mlsanity.com/paper/2410.18955/) (2024)
- [Surgical Action Planning with Large Language Models](https://mlsanity.com/paper/2503.18296/) (2025)
- [Schema as Parameterized Tools for Universal Information Extraction](https://mlsanity.com/paper/2506.01276/) (2025)
- [ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery](https://mlsanity.com/paper/2604.09237/) (2026)
- [Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Framework](https://mlsanity.com/paper/2508.12257/) (2025)
- [Evaluating Structured Decoding for Text-to-Table Generation: Evidence from Three Datasets](https://mlsanity.com/paper/2508.15910/) (2025)
- [DTBench: A Synthetic Benchmark for Document-to-Table Extraction](https://mlsanity.com/paper/2602.13812/) (2026)
- [Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis](https://mlsanity.com/paper/2502.03843/) (2025)
- [An agentic system with reinforcement-learned subsystem improvements for parsing form-like documents](https://mlsanity.com/paper/2505.13504/) (2025)
- [Zero-Shot Open-Schema Entity Structure Discovery](https://mlsanity.com/paper/2506.04458/) (2025)
- [ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images](https://mlsanity.com/paper/2602.12203/) (2026)
- [Instructor-Aligned Knowledge Graphs for Personalized Learning](https://mlsanity.com/paper/2602.17111/) (2026)
- [GRASP: GRanularity-Aware Search Policy for Agentic RAG](https://mlsanity.com/paper/2607.10463/) (2026)
- [MetaKP: On-Demand Keyphrase Generation](https://mlsanity.com/paper/2407.00191/) (2024)
- [Low-Resource Fine-Tuning for Multi-Task Structured Information Extraction with a Billion-Parameter Instruction-Tuned Model](https://mlsanity.com/paper/2509.08381/) (2025)
- [The Answer Lies Within: Self-Derived Rewards Enable Explainable Relation Extraction](https://mlsanity.com/paper/2510.06198/) (2025)
- [ToMMeR -- Efficient Entity Mention Detection from Large Language Models](https://mlsanity.com/paper/2510.19410/) (2025)
- [LiveWeb-IE: A Benchmark For Online Web Information Extraction](https://mlsanity.com/paper/2603.13773/) (2026)
- [Relational Probing: LM-to-Graph Adaptation for Financial Prediction](https://mlsanity.com/paper/2604.10212/) (2026)
- [Closing the Loop: Unified 3D Scene Generation and Immersive Interaction via LLM-RL Coupling](https://mlsanity.com/paper/2605.05711/) (2026)

## Abstract (arXiv)

Large language models with instruction-following capabilities open the door to a wider group of users. However, when it comes to information extraction - a classic task in natural language processing - most task-specific systems cannot align well with long-tail ad hoc extraction use cases for non-expert users. To address this, we propose a novel paradigm, termed On-Demand Information Extraction, to fulfill the personalized demands of real-world users. Our task aims to follow the instructions to extract the desired content from the associated text and present it in a structured tabular format. The table headers can either be user-specified or inferred contextually by the model. To facilitate research in this emerging area, we present a benchmark named InstructIE, inclusive of both automatically generated training data, as well as the human-annotated test set. Building on InstructIE, we further develop an On-Demand Information Extractor, ODIE. Comprehensive evaluations on our benchmark reveal that ODIE substantially outperforms the existing open-source models of similar size. Our code and dataset are released on https://github.com/yzjiao/On-Demand-IE.