# Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains

arXiv: https://arxiv.org/abs/2402.05140 · HTML: https://mlsanity.com/paper/2402.05140/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [CoCon: A Self-Supervised Approach for Controlled Text Generation](https://mlsanity.com/paper/2006.03535/) (2020)
- [Cross-Modal Fine-Tuning: Align then Refine](https://mlsanity.com/paper/2302.05738/) (2023)
- [NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks](https://mlsanity.com/paper/2110.05668/) (2021)
- [DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generation](https://mlsanity.com/paper/2210.09551/) (2022)
- [Tailor: A Prompt-Based Approach to Attribute-Based Controlled Text Generation](https://mlsanity.com/paper/2204.13362/) (2022)
- [Efficient Architecture Search for Diverse Tasks](https://mlsanity.com/paper/2204.07554/) (2022)
- [Reprogramming Pretrained Language Models for Protein Sequence Representation Learning](https://mlsanity.com/paper/2301.02120/) (2023)
- [Otter-Knowledge: benchmarks of multimodal knowledge graph representation learning from different sources for drug discovery](https://mlsanity.com/paper/2306.12802/) (2023)

## Built on by (most cited first, 30 of 30)

- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [500xCompressor: Generalized Prompt Compression for Large Language Models](https://mlsanity.com/paper/2408.03094/) (2024)
- [CodePDE: An Inference Framework for LLM-driven PDE Solver Generation](https://mlsanity.com/paper/2505.08783/) (2025)
- [Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs](https://mlsanity.com/paper/2502.14830/) (2025)
- [Zeroth-Order Fine-Tuning of LLMs in Random Subspaces](https://mlsanity.com/paper/2410.08989/) (2024)
- [Specialized Foundation Models Struggle to Beat Supervised Baselines](https://mlsanity.com/paper/2411.02796/) (2024)
- [OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery](https://mlsanity.com/paper/2503.17604/) (2025)
- [Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey](https://mlsanity.com/paper/2403.01528/) (2024)
- [Mixture of Structural-and-Textual Retrieval over Text-rich Graph Knowledge Bases](https://mlsanity.com/paper/2502.20317/) (2025)
- [TagRouter: Learning Route to LLMs through Tags for Open-Domain Text Generation Tasks](https://mlsanity.com/paper/2506.12473/) (2025)
- [A Large Language Model-Empowered Agent for Reliable and Robust Structural Analysis](https://mlsanity.com/paper/2507.02938/) (2025)
- [Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models](https://mlsanity.com/paper/2408.05241/) (2024)
- [Multi-Document Financial Question Answering using LLMs](https://mlsanity.com/paper/2411.07264/) (2024)
- [LoRA-Gen: Specializing Large Language Model via Online LoRA Generation](https://mlsanity.com/paper/2506.11638/) (2025)
- [A Multi-Task Evaluation of LLMs' Processing of Academic Text Input](https://mlsanity.com/paper/2508.11779/) (2025)
- [Generative Design of Functional Metal Complexes Utilizing the Internal Knowledge of Large Language Models](https://mlsanity.com/paper/2410.18136/) (2024)
- [PhenoAssistant: A Conversational Multi-Agent AI System for Automated Plant Phenotyping](https://mlsanity.com/paper/2504.19818/) (2025)
- [Autoware.Flex: Human-Instructed Dynamically Reconfigurable Autonomous Driving Systems](https://mlsanity.com/paper/2412.16265/) (2024)
- [MiST: Understanding the Role of Mid-Stage Scientific Training in Developing Chemical Reasoning Models](https://mlsanity.com/paper/2512.21231/) (2025)
- [Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers](https://mlsanity.com/paper/2506.14702/) (2025)
- [A Survey for Large Language Models in Biomedicine](https://mlsanity.com/paper/2409.00133/) (2024)
- [An Open-Source Dual-Loss Embedding Model for Semantic Retrieval in Higher Education](https://mlsanity.com/paper/2505.04916/) (2025)
- [Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning](https://mlsanity.com/paper/2511.07198/) (2025)
- [A Structure-Agnostic Co-Tuning Framework for LLMs and SLMs in Cloud-Edge Systems](https://mlsanity.com/paper/2511.11678/) (2025)
- [Do We Need Bigger Models for Science? Task-Aware Retrieval with Small Language Models](https://mlsanity.com/paper/2604.01965/) (2026)
- [General Intelligence-based Fragmentation (GIF): A framework for peak-labeled spectra simulation](https://mlsanity.com/paper/2511.09571/) (2025)
- [AbAffinity: A Large Language Model for Predicting Antibody Binding Affinity against SARS-CoV-2](https://mlsanity.com/paper/2603.04480/) (2026)
- [LegalMidm: Use-Case-Driven Legal Domain Specialization for Korean Large Language Model](https://mlsanity.com/paper/2604.25297/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable proficiency in understanding and generating natural language. However, their capabilities wane in highly specialized domains underrepresented in the pretraining corpus, such as physical and biomedical sciences. This work explores how to repurpose general LLMs into effective task solvers for specialized domains. We introduce a novel, model-agnostic framework for learning custom input tags, which are parameterized as continuous vectors appended to the LLM's embedding layer, to condition the LLM. We design two types of input tags: domain tags are used to delimit specialized representations (e.g., chemical formulas) and provide domain-relevant context; function tags are used to represent specific functions (e.g., predicting molecular properties) and compress function-solving instructions. We develop a three-stage protocol to learn these tags using auxiliary data and domain knowledge. By explicitly disentangling task domains from task functions, our method enables zero-shot generalization to unseen problems through diverse combinations of the input tags. It also boosts LLM's performance in various specialized domains, such as predicting protein or chemical properties and modeling drug-target interactions, outperforming expert models tailored to these tasks.