# Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling

arXiv: https://arxiv.org/abs/1910.04289 · HTML: https://mlsanity.com/paper/1910.04289/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 22 of 22)

- [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://mlsanity.com/paper/2103.08541/) (2021)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data](https://mlsanity.com/paper/2106.08977/) (2021)
- [Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation](https://mlsanity.com/paper/2105.09637/) (2021)
- [A Simple but Effective Approach to Improve Structured Language Model Output for Information Extraction](https://mlsanity.com/paper/2402.13364/) (2024)
- [BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition](https://mlsanity.com/paper/2105.12848/) (2021)
- [Mix of Experts Language Model for Named Entity Recognition](https://mlsanity.com/paper/2404.19192/) (2024)
- [Understanding Programmatic Weak Supervision via Source-aware Influence Function](https://mlsanity.com/paper/2205.12879/) (2022)
- [Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification](https://mlsanity.com/paper/2210.03092/) (2022)
- [Truth Discovery in Sequence Labels from Crowds](https://mlsanity.com/paper/2109.04470/) (2021)
- [Prompt to be Consistent is Better than Self-Consistent? Few-Shot and Zero-Shot Fact Verification with Pre-trained Language Models](https://mlsanity.com/paper/2306.02569/) (2023)
- [Sparse Conditional Hidden Markov Model for Weakly Supervised Named Entity Recognition](https://mlsanity.com/paper/2205.14228/) (2022)
- [Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labeler](https://mlsanity.com/paper/2309.05086/) (2023)
- [Binary Classification with Positive Labeling Sources](https://mlsanity.com/paper/2208.01704/) (2022)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Few-shot Named Entity Recognition with Cloze Questions](https://mlsanity.com/paper/2111.12421/) (2021)
- [ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision](https://mlsanity.com/paper/2204.06863/) (2022)
- [WeShap: Weak Supervision Source Evaluation with Shapley Values](https://mlsanity.com/paper/2406.11010/) (2024)
- [Automating Weak Label Generation for Data Programming with Clinicians in the Loop](https://mlsanity.com/paper/2407.07982/) (2024)
- [Modeling sequential annotations for sequence labeling with crowds](https://mlsanity.com/paper/2209.09430/) (2022)
- [Utilizing Weak Supervision To Generate Indonesian Conservation Dataset](https://mlsanity.com/paper/2310.11258/) (2023)

## Abstract (arXiv)

Sequence labeling is a fundamental framework for various natural language processing problems. Its performance is largely influenced by the annotation quality and quantity in supervised learning scenarios, and obtaining ground truth labels is often costly. In many cases, ground truth labels do not exist, but noisy annotations or annotations from different domains are accessible. In this paper, we propose a novel framework Consensus Network (ConNet) that can be trained on annotations from multiple sources (e.g., crowd annotation, cross-domain data...). It learns individual representation for every source and dynamically aggregates source-specific knowledge by a context-aware attention module. Finally, it leads to a model reflecting the agreement (consensus) among multiple sources. We evaluate the proposed framework in two practical settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Extensive experimental results show that our model achieves significant improvements over existing methods in both settings. We also demonstrate that the method can apply to various tasks and cope with different encoders.