mlsanityData to 7 Oct 2026

LMPriors: Pre-Trained Language Models as Task-Specific Priors

arXiv 2210.12530 ↗First posted 22 Oct 2022cs.LG cs.AI cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

Particularly in low-data regimes, an outstanding challenge in machine learning is developing principled techniques for augmenting our models with suitable priors. This is to encourage them to learn in ways that are compatible with our understanding of the world. But in contrast to generic priors such as shrinkage or sparsity, we draw inspiration from the recent successes of large-scale language models (LMs) to construct task-specific priors distilled from the rich knowledge of LMs. Our method, Language Model Priors (LMPriors), incorporates auxiliary natural language metadata about the task -- such as variable names and descriptions -- to encourage downstream model outputs to be consistent with the LM's common-sense reasoning based on the metadata. Empirically, we demonstrate that LMPriors improve model performance in settings where such natural language descriptions are available, and perform well on several tasks that benefit from such prior knowledge, such as feature selection, causal inference, and safe reinforcement learning.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 39 of its 67 references that are in the map. First posted on arXiv 22 Oct 2022.

Builds on

39 of 67 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
27 more
  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2018Datasheets for Datasetscited by 1,371
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  6. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  11. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 459
  12. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 443
  13. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  14. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  15. 2021Skillful Precipitation Nowcasting using Deep Generative Models of Radarcited by 308
  16. 2019Adversarial Examples Improve Image Recognitioncited by 283
  17. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  18. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  19. 2020MetNet: A Neural Weather Model for Precipitation Forecastingcited by 128
  20. 2017AI Safety Gridworldscited by 108
  21. 2019LassoNet: A Neural Network with Feature Sparsitycited by 81
  22. 2014Distinguishing cause from effect using observational data: methods and benchmarkscited by 33
  23. 2021No-Press Diplomacy from Scratchcited by 31
  24. 2021Frequency-based Distortions in Contextualized Word Embeddingscited by 19
  25. 2017The prior can generally only be understood in the context of the likelihoodcited by 18
  26. 2020Causal Mosaic: Cause-Effect Inference via Nonlinear ICA and Ensemble Methodcited by 16
  27. 2014Penalising model component complexity: A principled, practical approach to constructing priorscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 15; 2025: 17; 2026: 12.

56 papers.

Sort:

By area:

  1. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  2. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  3. 2023Causal Discovery with Language Models as Imperfect Expertscited by 62
  4. 2023Integrating Large Language Model for Improved Causal Discoverycited by 54
  5. 2024Efficient Causal Graph Discovery Using Large Language Modelscited by 51
  6. 2023Causal Order: The Key to Leveraging Imperfect Experts in Causal Inferencecited by 46
  7. 2023Prompting is not a substitute for probability measurements in large language modelscited by 43
  8. 2024Exploring Large Language Models for Feature Selection: A Data-centric Perspectivecited by 27
  9. 2024Multi-Agent Causal Discovery Using Large Language Modelscited by 26
  10. 2023Zero-Shot Robustification of Zero-Shot Modelscited by 24
  11. 2024Large Language Models are Effective Priors for Causal Graph Discoverycited by 23
  12. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
38 more
  1. 2024Causal Graph Discovery with Retrieval-Augmented Generation based Large Language Modelscited by 16
  2. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
  3. 2024AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modellingcited by 15
  4. 2025Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discoverycited by 15
  5. 2023LaMPP: Language Models as Probabilistic Priors for Perception and Actioncited by 11
  6. 2023Emerging Synergies in Causality and Deep Generative Models: A Surveycited by 10
  7. 2023AdaRefiner: Refining Decisions of Language Models with Adaptive Feedbackcited by 10
  8. 2024Discovering and Reasoning of Causality in the Hidden World with Large Language Modelscited by 10
  9. 2025LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularizationcited by 10
  10. 2024NICE: To Optimize In-Context Examples or Not?cited by 9
  11. 2024Knowledge-Driven Feature Selection and Engineering for Genotype Data with Large Language Modelscited by 9
  12. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrationscited by 7
  13. 2024World Models with Hints of Large Language Models for Goal Achievingcited by 6
  14. 2023LaGR-SEQ: Language-Guided Reinforcement Learning with Sample-Efficient Queryingcited by 5
  15. 2025Fairness-Driven LLM-based Causal Discovery with Active Learning and Dynamic Scoringcited by 5
  16. 2025No Black Boxes: Interpretable and Interactable Predictive Healthcare with Knowledge-Enhanced Agentic Causal Discoverycited by 5
  17. 2025Domain-Aware Tensor Network Structure Searchcited by 5
  18. 2025Bayesian Network Structure Discovery Using Large Language Modelscited by 5
  19. 2025HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Drivingcited by 4
  20. 2024Towards LLM-guided Efficient and Interpretable Multi-linear Tensor Network Rank Selectioncited by 3
  21. 2024MedG-KRP: Medical Graph Knowledge Representation Probingcited by 2
  22. 2025Learning to Defer for Causal Discovery with Imperfect Expertscited by 2
  23. 2025LLM4FS: Leveraging Large Language Models for Feature Selectioncited by 2
  24. 2025Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLMcited by 2
  25. 2025Causal MAS: A Survey of Large Language Model Architectures for Discovery and Effect Estimationcited by 2
  26. 2026LLMs can construct powerful representations and streamline sample-efficient supervised learningcited by 2
  27. 2024Preferential Normalizing Flowscited by 1
  28. 2025Exploiting LLMs for Automatic Hypothesis Assessment via a Logit-Based Calibrated Priorcited by 1
  29. 2025Multi-level Diagnosis and Evaluation for Robust Tabular Feature Engineering with Large Language Modelscited by 1
  30. 2025ClinStructor: AI-Powered Structuring of Unstructured Clinical Textscited by 1
  31. 2026LLM Flow Processes for Text-Conditioned Regressioncited by 1
  32. 2026CASSANDRA: Programmatic and Probabilistic Learning and Inference for Stochastic World Modelingcited by 1
  33. 2026LLM-Driven Reasoning for Constraint-Aware Feature Selection in Industrial Systemscited by 1
  34. 2023Can We Utilize Pre-trained Language Models within Causal Discovery Algorithms?cited by 0
  35. 2025Accelerating Causal Network Discovery of Alzheimer Disease Biomarkers via Scientific Literature-based Retrieval Augmented Generationcited by 0
  36. 2025DeepFeature: LLM-Empowered Context-aware Feature Generation for Wearable Biosignalscited by 0
  37. 2026ReToP: Learning to Rewrite Electronic Health Records for Clinical Predictioncited by 0
  38. 2026Learning When to Trust LLM Priors: A Validated Framework for Semantic Prior Integrationcited by 0

50 most cited of 56 All 56 (JSON)