# Distilling Task-specific Logical Rules from Large Pre-trained Models

arXiv: https://arxiv.org/abs/2210.02768 · HTML: https://mlsanity.com/paper/2210.02768/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021)
- [Automated Phrase Mining from Massive Text Corpora](https://mlsanity.com/paper/1702.04457/) (2017)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [Learning the Structure of Generative Models without Labeled Data](https://mlsanity.com/paper/1703.00854/) (2017)
- [Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning](https://mlsanity.com/paper/1906.01378/) (2019)
- [Named Entity Recognition without Labelled Data: A Weak Supervision Approach](https://mlsanity.com/paper/2004.14723/) (2020)
- [Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model](https://mlsanity.com/paper/2106.04098/) (2021)
- [TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition](https://mlsanity.com/paper/2004.07493/) (2020)
- [SwellShark: A Generative Model for Biomedical Named Entity Recognition without Labeled Data](https://mlsanity.com/paper/1704.06360/) (2017)
- [Empower Entity Set Expansion via Language Model Probing](https://mlsanity.com/paper/2004.13897/) (2020)
- [Generalizing Natural Language Analysis through Span-relation Representations](https://mlsanity.com/paper/1911.03822/) (2019)
- [Weakly Supervised Named Entity Tagging with Learnable Logical Rules](https://mlsanity.com/paper/2107.02282/) (2021)
- [Unsupervised Aspect Term Extraction with B-LSTM & CRF using Automatically Labelled Datasets](https://mlsanity.com/paper/1709.05094/) (2017)

## Built on by (most cited first, 1 of 1)

- [Prompt Engineering for Healthcare: Methodologies and Applications](https://mlsanity.com/paper/2304.14670/) (2023)

## Abstract (arXiv)

Logical rules, both transferable and explainable, are widely used as weakly supervised signals for many downstream tasks such as named entity tagging. To reduce the human effort of writing rules, previous researchers adopt an iterative approach to automatically learn logical rules from several seed rules. However, obtaining more seed rules can only be accomplished by extra human annotation with heavy costs. Limited by the size and quality of the seed rules, the model performance of previous systems is bounded. In this paper, we develop a novel framework STREAM to distill task-specific logical rules from large pre-trained models. Specifically, we borrow recent prompt-based language models as the knowledge expert to yield initial seed rules, and based on the formed high-quality instance pool that acts as an intermediary role, we keep teaching the expert to fit our task and learning task-specific logical rules. Experiments on three public named entity tagging benchmarks demonstrate the effectiveness of our proposed framework. With several predefined prompt templates, our system has gained significant improvements over previous state-of-the-art methods.