mlsanityData to 7 Oct 2026

Semi-supervised Sequence Learning

arXiv 1511.01432 ↗First posted 4 Nov 2015cs.LG cs.CLcited by 198

Not on the map. Why?

Abstract (arXiv)

We present two approaches that use unlabeled data to improve sequence learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a conventional language model in natural language processing. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a "pretraining" step for a later supervised sequence learning algorithm. In other words, the parameters obtained from the unsupervised step can be used as a starting point for other supervised training models. In our experiments, we find that long short term memory recurrent networks after being pretrained with the two approaches are more stable and generalize better. With pretraining, we are able to train long short term memory recurrent networks up to a few hundred timesteps, thereby achieving strong performance in many text classification tasks, such as IMDB, DBpedia and 20 Newsgroups.

Key facts. Cited by 198 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 2; 2021: 62; 2022: 47; 2023: 32; 2024: 34; 2025: 11; 2026: 8.

198 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2023PaLM 2 Technical Reportcited by 1,154
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  12. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
38 more
  1. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  2. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  4. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  5. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  6. 2022UL2: Unifying Language Learning Paradigmscited by 292
  7. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  8. 2021Scaling Laws for Transfercited by 265
  9. 2021A Survey on Deep Semi-supervised Learningcited by 250
  10. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  11. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  12. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  13. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  14. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  15. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  16. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  17. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  18. 2021AP-10K: A Benchmark for Animal Pose Estimation in the Wildcited by 123
  19. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  20. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  21. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  22. 2022USB: A Unified Semi-supervised Learning Benchmark for Classificationcited by 95
  23. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  24. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  25. 2022MetaMorph: Learning Universal Controllers with Transformerscited by 78
  26. 2023MOTOR: A Time-To-Event Foundation Model For Structured Medical Recordscited by 75
  27. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  28. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 58
  29. 2022OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regressioncited by 44
  30. 2022Grad-SAM: Explaining Transformers via Gradient Self-Attention Mapscited by 42
  31. 2023Fractional Denoising for 3D Molecular Pre-trainingcited by 33
  32. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  33. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 33
  34. 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
  35. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  36. 2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
  37. 2021The Rich Get Richer: Disparate Impact of Semi-Supervised Learningcited by 28
  38. 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27

50 most cited of 198 All 198 (JSON)