mlsanityData to 7 Oct 2026

SURGELLM: Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization

arXiv 2606.24259 ↗First posted 23 Jun 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Fine-tuned encoders deployed across heterogeneous NLP tasks face three compounding problems: mismatched inductive biases, class-imbalance corruption of feature statistics, and no mechanism to condition attention on external lexical knowledge. We introduce surgellm, a unified transformer framework that addresses each with a dedicated lightweight module: a surgical feature gate (learned per-dimension sigmoid over curated lexical indicators and texttt[CLS]; provably degenerates to identity when features are uninformative), task-conditioned prefix tokens (quantized feature values and task identity prepended to every input), and Instance-Weighted Normalization (IWN; removes class-prior bias from gate statistics). We prove an excess-risk bound linking gate benefit to surgical feature alignment. Across four tasks, SST-2, multi-hop retrieval, LLM-prompt attribution, and authorship detection, covering 17,830 examples and eleven model variants over three seeds, the IWN variant achieves macro-F1 0.940 (+0.036 over the strongest non-IWN baseline; +0.130 on authorship detection). A random-vocabulary control (-0.028 avg. F1) confirms gains are lexical, not parametric. Code, vocabularies, and a 99.5%-recovery auto-extraction recipe are released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 13 of its 48 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

13 of 48 references.

References · 13

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2016Layer Normalizationcited by 5,579
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  7. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  8. 2023A Watermark for Large Language Modelscited by 667
  9. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  10. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
  11. 2015Training Very Deep Networkscited by 172
  12. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 113
1 more
  1. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Task-Aware LoRA Adapter Composition via Similarity Retrieval in Vector Databases
  2. 2024Surgical Feature-Space Decomposition of LLMs: Why, When and How?
  3. 2026DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models
  4. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
  5. 2024On the Power of Convolution Augmented Transformer
  6. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts
  7. 2026UNMASK: Discovering and Causally Verifying Spurious Shortcuts in Text Classifiers
  8. 2026Measuring Semantic Abstractness of SAE Features via Nonlocality