mlsanityData to 7 Oct 2026

Learned in Translation: Contextualized Word Vectors

arXiv 1708.00107 ↗First posted 1 Aug 2017cs.CL cs.AI cs.LGcited by 104

Not on the map. Why?

Abstract (arXiv)

Computer vision has benefited from initializing multiple deep layers with weights pretrained on large supervised training sets like ImageNet. Natural language processing (NLP) typically sees initialization of only the lowest layer of deep models with pretrained word vectors. In this paper, we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation (MT) to contextualize word vectors. We show that adding these context vectors (CoVe) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks: sentiment analysis (SST, IMDb), question classification (TREC), entailment (SNLI), and question answering (SQuAD). For fine-grained sentiment analysis and entailment, CoVe improves performance of our baseline models to the state of the art.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 46; 2022: 32; 2023: 10; 2024: 10; 2025: 2; 2026: 3.

104 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  4. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  5. 2021Dive into Deep Learningcited by 111
  6. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  7. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 61
  8. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  9. 2022Pretraining Without Attentioncited by 51
  10. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
  11. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 40
  12. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
38 more
  1. 2021Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attentioncited by 36
  2. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  3. 2021Conversational Question Answering: A Surveycited by 31
  4. 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27
  5. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  6. 2021Fully Transformer Networks for Semantic Image Segmentationcited by 24
  7. 2021Czert -- Czech BERT-like Model for Language Representationcited by 22
  8. 2023Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applicationscited by 21
  9. 2024Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challengescited by 20
  10. 2022SCD: Self-Contrastive Decorrelation for Sentence Embeddingscited by 19
  11. 2022TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalitiescited by 19
  12. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  13. 2025Scaling Embedding Layers in Language Modelscited by 19
  14. 2021Rethinking Search: Making Domain Experts out of Dilettantescited by 18
  15. 2021The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architecturescited by 16
  16. 2021Student Surpasses Teacher: Imitation Attack for Black-Box NLP APIscited by 16
  17. 2022MVP: Multi-task Supervised Pre-training for Natural Language Generationcited by 16
  18. 2022Multi-task Active Learning for Pre-trained Transformer-based Modelscited by 16
  19. 2021DOCENT: Learning Self-Supervised Entity Representations from Large Document Collectionscited by 13
  20. 2021MergeDistill: Merging Pre-trained Language Models using Distillationcited by 13
  21. 2021Diverse Pretrained Context Encodings Improve Document Translationcited by 13
  22. 2021On the Use of External Data for Spoken Named Entity Recognitioncited by 12
  23. 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learningcited by 12
  24. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  25. 2026Shaping capabilities with token-level data filteringcited by 11
  26. 2024Financial Sentiment Analysis on News and Reports Using Large Language Models and FinBERTcited by 9
  27. 2021Outline to Story: Fine-grained Controllable Story Generation from Cascaded Eventscited by 8
  28. 2022Structure Extraction in Task-Oriented Dialogues with Slot Clusteringcited by 8
  29. 2022Federated Pruning: Improving Neural Network Efficiency with Federated Learningcited by 8
  30. 2021The Rediscovery Hypothesis: Language Models Need to Meet Linguisticscited by 7
  31. 2021PVG at WASSA 2021: A Multi-Input, Multi-Task, Transformer-Based Architecture for Empathy and Distress Predictioncited by 7
  32. 2021Language Model Augmented Relevance Scorecited by 7
  33. 2021Are the Multilingual Models Better? Improving Czech Sentiment with Transformerscited by 7
  34. 2021Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Researchcited by 7
  35. 2021Introduction to Neural Transfer Learning with Transformers for Social Science Text Analysiscited by 6
  36. 2021Enhance Long Text Understanding via Distilled Gist Detector from Abstractive Summarizationcited by 6
  37. 2022Empirical Evaluation and Theoretical Analysis for Representation Learning: A Surveycited by 5
  38. 2022Socially Enhanced Situation Awareness from Microblogs using Artificial Intelligence: A Surveycited by 5

50 most cited of 104 All 104 (JSON)