# A Cross-Task Analysis of Text Span Representations

arXiv: https://arxiv.org/abs/2006.03866 · HTML: https://mlsanity.com/paper/2006.03866/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2020.

## Built on by (most cited first, 21 of 21)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration](https://mlsanity.com/paper/2109.06304/) (2021)
- [Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding](https://mlsanity.com/paper/2210.03304/) (2022)
- [Few-shot Intent Classification and Slot Filling with Retrieved Examples](https://mlsanity.com/paper/2104.05763/) (2021)
- [Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representations](https://mlsanity.com/paper/2109.05958/) (2021)
- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)
- [On the Interplay Between Fine-tuning and Composition in Transformers](https://mlsanity.com/paper/2105.14668/) (2021)
- [Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing](https://mlsanity.com/paper/2110.08538/) (2021)
- [Augmenting Transformers with Recursively Composed Multi-grained Representations](https://mlsanity.com/paper/2309.16319/) (2023)
- [A Controlled Reevaluation of Coreference Resolution Models](https://mlsanity.com/paper/2404.00727/) (2024)
- [Win-Win Cooperation: Bundling Sequence and Span Models for Named Entity Recognition](https://mlsanity.com/paper/2207.03300/) (2022)
- [COMBO: A Complete Benchmark for Open KG Canonicalization](https://mlsanity.com/paper/2302.03905/) (2023)
- [Learning Language Structures through Grounding](https://mlsanity.com/paper/2406.09662/) (2024)
- [MaxPoolBERT: Enhancing BERT Classification via Layer- and Token-Wise Aggregation](https://mlsanity.com/paper/2505.15696/) (2025)
- [Span Fine-tuning for Pre-trained Language Models](https://mlsanity.com/paper/2108.12848/) (2021)
- [MuLMS: A Multi-Layer Annotated Text Corpus for Information Extraction in the Materials Science Domain](https://mlsanity.com/paper/2310.15569/) (2023)
- [Confidence-Aware Sub-Structure Beam Search (CABS): Mitigating Hallucination in Structured Data Generation with Large Language Models](https://mlsanity.com/paper/2406.00069/) (2024)
- [Towards Proactively Forecasting Sentence-Specific Information Popularity within Online News Documents](https://mlsanity.com/paper/2301.00152/) (2022)
- [Dynamic Named Entity Recognition](https://mlsanity.com/paper/2302.10314/) (2023)
- [Impact of Subword Pooling Strategy on Cross-lingual Event Detection](https://mlsanity.com/paper/2302.11365/) (2023)
- [A Novel Prompt-tuning Method: Incorporating Scenario-specific Concepts into a Verbalizer](https://mlsanity.com/paper/2401.05204/) (2024)

## Abstract (arXiv)

Many natural language processing (NLP) tasks involve reasoning with textual spans, including question answering, entity recognition, and coreference resolution. While extensive research has focused on functional architectures for representing words and sentences, there is less work on representing arbitrary spans of text within sentences. In this paper, we conduct a comprehensive empirical evaluation of six span representation methods using eight pretrained language representation models across six tasks, including two tasks that we introduce. We find that, although some simple span representations are fairly reliable across tasks, in general the optimal span representation varies by task, and can also vary within different facets of individual tasks. We also find that the choice of span representation has a bigger impact with a fixed pretrained encoder than with a fine-tuned encoder.