mlsanityData to 7 Oct 2026

LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts

arXiv 2602.14060 ↗First posted 15 Feb 2026cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We introduce LM-Lexicon, an innovative definition modeling approach that incorporates data clustering, semantic expert learning, and model merging using a sparse mixture-of-experts architecture. By decomposing the definition modeling task into specialized semantic domains, where small language models are trained as domain experts, LM-Lexicon achieves substantial improvements (+7% BLEU score compared with the prior state-of-the-art model) over existing methods on five widely used benchmarks. Empirically, we demonstrate that 1) the clustering strategy enables fine-grained expert specialization with nearly 10% improvement in definition quality; 2) the semantic-aware domain-level routing mechanism achieves higher expert efficacy (+1%) than conventional token-level routing; and 3) further performance gains can be obtained through test-time compute and semantic expert scaling. Our work advances definition modeling while providing insights into the development of efficient language models for semantic-intensive applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 34 of its 61 references that are in the map. First posted on arXiv 15 Feb 2026.

Builds on

34 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs3 Feb
  2. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training24 Jun
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  4. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar
  5. 2024ConstitutionalExperts: Training a Mixture of Principle-based Prompts7 Mar
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  7. 2023Task-Based MoE for Multitask Multilingual Machine Translation30 Aug
  8. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 25

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  8. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  9. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  10. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  11. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  12. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
13 more
  1. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  2. 2016Definition Modeling: Learning to define word embeddings in natural languagecited by 46
  3. 2024Instruction Tuning With Loss Over Instructionscited by 40
  4. 2018Conditional Generators of Words Definitionscited by 35
  5. 2024MoDE: CLIP Data Experts via Clusteringcited by 22
  6. 2017Learning to Explain Non-Standard English Words and Phrasescited by 21
  7. 2023Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysiscited by 19
  8. 2022Multitasking Framework for Unsupervised Simple Definition Generationcited by 18
  9. 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMscited by 12
  10. 2025ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflectioncited by 11
  11. 2022Fine-grained Contrastive Learning for Definition Generationcited by 10
  12. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  13. 2023Exploiting Correlations Between Contexts and Definitions with Multiple Definition Modelingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Latent Semantic Clustering for Scaling Test-Time Computation of LLMs
  2. 2025How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
  3. 2025Probing Semantic Routing in Large Mixture-of-Expert Models
  4. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
  5. 2024Unconstrained Model Merging for Enhanced LLM Reasoning
  6. 2026Do Domain-specific Experts exist in MoE-based LLMs?
  7. 2026Mixture of Experts for Low-Resource LLMs
  8. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clustering