# Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens

arXiv: https://arxiv.org/abs/1805.02214 · HTML: https://mlsanity.com/paper/1805.02214/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2018.

## Built on by (most cited first, 12 of 12)

- [On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency](https://mlsanity.com/paper/2203.07559/) (2022)
- [Supervising Model Attention with Human Explanations for Robust Natural Language Inference](https://mlsanity.com/paper/2104.08142/) (2021)
- [Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold](https://mlsanity.com/paper/2206.09755/) (2022)
- [Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models](https://mlsanity.com/paper/2205.11432/) (2022)
- [Atomic Inference for NLI with Generated Facts as Atoms](https://mlsanity.com/paper/2305.13214/) (2023)
- [Few-shot Named Entity Recognition with Self-describing Networks](https://mlsanity.com/paper/2203.12252/) (2022)
- [Learning from Language Description: Low-shot Named Entity Recognition via Decomposed Framework](https://mlsanity.com/paper/2109.05357/) (2021)
- [Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers](https://mlsanity.com/paper/2103.14465/) (2021)
- [Tell Model Where to Attend: Improving Interpretability of Aspect-Based Sentiment Classification via Small Explanation Annotations](https://mlsanity.com/paper/2302.10479/) (2023)
- [Utilizing coarse-grained data in low-data settings for event extraction](https://mlsanity.com/paper/2205.05468/) (2022)
- [Finding the Needle in a Haystack: Unsupervised Rationale Extraction from Long Text Classifiers](https://mlsanity.com/paper/2303.07991/) (2023)
- [Evaluating and explaining training strategies for zero-shot cross-lingual news sentiment analysis](https://mlsanity.com/paper/2409.20054/) (2024)

## Abstract (arXiv)

Can attention- or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level labels? We construct a neural network architecture based on soft attention, train it as a binary sentence classifier and evaluate against token-level annotation on four different datasets. Inferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating useful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, sometimes even rivaling the supervised oracle network.