# Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Models

arXiv: https://arxiv.org/abs/2609.13556 · HTML: https://mlsanity.com/paper/2609.13556/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 11 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [UltraMedical: Building Specialized Generalists in Biomedicine](https://mlsanity.com/paper/2406.03949/) (2024)
- [A Weighted Correlation Index for Rankings with Ties](https://mlsanity.com/paper/1404.3325/) (2014)
- [Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering](https://mlsanity.com/paper/2305.11541/) (2023)
- [MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling](https://mlsanity.com/paper/2305.08264/) (2023)
- [Multiple-Choice Questions are Efficient and Robust LLM Evaluators](https://mlsanity.com/paper/2405.11966/) (2024)
- [MedJEx: A Medical Jargon Extraction Model with Wiki's Hyperlink Span and Contextualized Masked Language Model Score](https://mlsanity.com/paper/2210.05875/) (2022)
- [README: Bridging Medical Jargon and Lay Understanding for Patient Education through Data-Centric NLP](https://mlsanity.com/paper/2312.15561/) (2023)
- [Understanding Jargon: Combining Extraction and Generation for Definition Modeling](https://mlsanity.com/paper/2111.07267/) (2021)
- [Words as Gatekeepers: Measuring Discipline-specific Terms and Meanings in Scholarly Publications](https://mlsanity.com/paper/2212.09676/) (2022)
- [Personalized Jargon Identification for Enhanced Interdisciplinary Communication](https://mlsanity.com/paper/2311.09481/) (2023)
- [Graphine: A Dataset for Graph-aware Terminology Definition Generation](https://mlsanity.com/paper/2109.04018/) (2021)
- [When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models](https://mlsanity.com/paper/2406.13131/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable proficiency on general-purpose tasks, yet their performance often degrades in highly-specialized technical domains. Moreover, little is known about how parametric knowledge of domain-specific terms is encoded within these models. We address this gap by contributing two novel medical jargon evaluation benchmarks and evaluate a general-purpose Llama-3.1 model against a variant fine-tuned on medical-domain data. Surprisingly, the general-purpose model outperforms the medically fine-tuned model on both tasks. Using mechanistic interpretability tools, we find systematic patterns of miscalibration for the medically fine-tuned model. Instead of reorganizing parametric knowledge, the fine-tuned model places greater emphasis on a small subset of model components associated with jargon-favoring predictions. We find that applying component reweighting strategies against the benchmark tasks successfully suppresses these components and closes the gap with the general-purpose baseline. We also observe that some jargon-sensitive components transfer knowledge to the same tasks involving materials science jargon, suggesting they encode a partially domain-agnostic notion of specialized terminology. Our results provide a case study in which a medically fine-tuned checkpoint does not improve jargon comprehension over its general-purpose counterpart, highlighting that domain adaptation should not be assumed to yield better performance on specialized terminology.