mlsanityData to 7 Oct 2026

Analysis and Prediction of NLP Models Via Task Embeddings

arXiv 2112.05647 ↗First posted 10 Dec 2021cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Task embeddings are low-dimensional representations that are trained to capture task properties. In this paper, we propose MetaEval, a collection of 101 NLP tasks. We fit a single transformer to all MetaEval tasks jointly while conditioning it on learned embeddings. The resulting task embeddings enable a novel analysis of the space of tasks. We then show that task aspects can be mapped to task embeddings for new tasks without using any annotated examples. Predicted embeddings can modulate the encoder for zero-shot inference and outperform a zero-shot baseline on GLUE tasks. The provided multitask setup can function as a benchmark for future transfer learning research.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 37 of its 53 references that are in the map. First posted on arXiv 10 Dec 2021.

Builds on

37 of 53 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2016Layer Normalizationcited by 5,579
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  12. 2016HyperNetworkscited by 1,003
25 more
  1. 2020Aligning AI With Shared Human Valuescited by 706
  2. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  3. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  4. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  5. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  6. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  7. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  8. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  9. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  10. 2019Continual learning with hypernetworkscited by 267
  11. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  12. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  13. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  14. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  15. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 113
  16. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  17. 2020Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!cited by 73
  18. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  19. 2020Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Datacited by 50
  20. 2019Spherical Text Embeddingcited by 49
  21. 2019Mining Discourse Markers for Unsupervised Sentence Representation Learningcited by 34
  22. 2018Factors Influencing the Surprising Instability of Word Embeddingscited by 31
  23. 2016How Transferable are Neural Networks in NLP Applications?cited by 23
  24. 2020Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)cited by 12
  25. 2020Modelling Latent Skills for Multitask Language Generationcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 2.

3 papers.

  1. 2022Joint processing of linguistic properties in brains and language modelscited by 37
  2. 2025Explainability in Practice: A Survey of Explainable NLP Across Various Domainscited by 11
  3. 2025MetaRank: Task-Aware Metric Selection for Model Transferability Estimationcited by 0