# Semi-Supervised Sequence Modeling with Cross-View Training

arXiv: https://arxiv.org/abs/1809.08370 · HTML: https://mlsanity.com/paper/1809.08370/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2018.

## Built on by (most cited first, 30 of 75)

- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)
- [A Unified Generative Framework for Various NER Subtasks](https://mlsanity.com/paper/2106.01223/) (2021)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://mlsanity.com/paper/2106.07499/) (2021)
- [USB: A Unified Semi-supervised Learning Benchmark for Classification](https://mlsanity.com/paper/2208.07204/) (2022)
- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [Prompt Consistency for Zero-Shot Task Generalization](https://mlsanity.com/paper/2205.00049/) (2022)
- [A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods](https://mlsanity.com/paper/2204.03508/) (2022)
- [Multi-view Subword Regularization](https://mlsanity.com/paper/2103.08490/) (2021)
- [Instruction Tuning With Loss Over Instructions](https://mlsanity.com/paper/2405.14394/) (2024)
- [Generative Artificial Intelligence: A Systematic Review and Applications](https://mlsanity.com/paper/2405.11029/) (2024)
- [Models and Datasets for Cross-Lingual Summarisation](https://mlsanity.com/paper/2202.09583/) (2022)
- [HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization](https://mlsanity.com/paper/2106.00149/) (2021)
- [The Rich Get Richer: Disparate Impact of Semi-Supervised Learning](https://mlsanity.com/paper/2110.06282/) (2021)
- [Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering](https://mlsanity.com/paper/2106.11575/) (2021)
- [Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP](https://mlsanity.com/paper/2110.03618/) (2021)
- [Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data](https://mlsanity.com/paper/2106.08977/) (2021)
- [Rethinking Semi-supervised Learning with Language Models](https://mlsanity.com/paper/2305.13002/) (2023)
- [Universal Self-Adaptive Prompting](https://mlsanity.com/paper/2305.14926/) (2023)
- [Dependency Parsing as MRC-based Span-Span Prediction](https://mlsanity.com/paper/2105.07654/) (2021)
- [Rethinking Negative Sampling for Handling Missing Entity Annotations](https://mlsanity.com/paper/2108.11607/) (2021)
- [ConNER: Consistency Training for Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2211.09394/) (2022)
- [FLamE: Few-shot Learning from Natural Language Explanations](https://mlsanity.com/paper/2306.08042/) (2023)
- [A Little Pretraining Goes a Long Way: A Case Study on Dependency Parsing Task for Low-resource Morphologically Rich Languages](https://mlsanity.com/paper/2102.06551/) (2021)
- [Consistency Training with Virtual Adversarial Discrete Perturbation](https://mlsanity.com/paper/2104.07284/) (2021)
- [A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021](https://mlsanity.com/paper/2106.04216/) (2021)
- [Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach](https://mlsanity.com/paper/2104.04886/) (2021)
- [Dual-View Distilled BERT for Sentence Embedding](https://mlsanity.com/paper/2104.08675/) (2021)
- [Progressive Class Semantic Matching for Semi-supervised Text Classification](https://mlsanity.com/paper/2205.10189/) (2022)
- [Something Old, Something New: Grammar-based CCG Parsing with Transformer Models](https://mlsanity.com/paper/2109.10044/) (2021)

## Abstract (arXiv)

Unsupervised representation learning algorithms such as word2vec and ELMo improve the accuracy of many supervised NLP models, mainly because they can take advantage of large amounts of unlabeled text. However, the supervised models only learn from task-specific labeled data during the main training phase. We therefore propose Cross-View Training (CVT), a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data. On labeled examples, standard supervised learning is used. On unlabeled examples, CVT teaches auxiliary prediction modules that see restricted views of the input (e.g., only part of a sentence) to match the predictions of the full model seeing the whole input. Since the auxiliary modules and the full model share intermediate representations, this in turn improves the full model. Moreover, we show that CVT is particularly effective when combined with multi-task learning. We evaluate CVT on five sequence tagging tasks, machine translation, and dependency parsing, achieving state-of-the-art results.