mlsanityData to 7 Oct 2026

Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers

arXiv 2103.14465 ↗First posted 26 Mar 2021cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 26 Mar 2021.

Builds on

20 of 29 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2019Attention is not Explanationcited by 916
  6. 2019Attention is not not Explanationcited by 629
  7. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  8. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  9. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  10. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  11. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  12. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
8 more
  1. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  2. 2017The Promise and Peril of Human Evaluation for Model Interpretabilitycited by 52
  3. 2019Generating Token-Level Explanations for Natural Language Inferencecited by 28
  4. 2018Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answeringcited by 23
  5. 2016Compositional Sequence Labeling Models for Error Detection in Learner Writingcited by 16
  6. 2019Context is Key: Grammatical Error Detection with Contextual Word Representationscited by 14
  7. 2018Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokenscited by 12
  8. 2018Jointly Learning to Label Sentences and Tokenscited by 9

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 1; 2023: 2.

5 papers.

  1. 2021Supervising Model Attention with Human Explanations for Robust Natural Language Inferencecited by 29
  2. 2021Translation Error Detection as Rationale Extractioncited by 22
  3. 2022Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Modelscited by 17
  4. 2023Atomic Inference for NLI with Generated Facts as Atomscited by 16
  5. 2023Finding the Needle in a Haystack: Unsupervised Rationale Extraction from Long Text Classifierscited by 1