# GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification

arXiv: https://arxiv.org/abs/2603.10008 · HTML: https://mlsanity.com/paper/2603.10008/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [[CLS] Token Tells Everything Needed for Training-free Efficient MLLMs](https://mlsanity.com/paper/2412.05819/) (2024)
- [Multi-Sample Dropout for Accelerated Training and Better Generalization](https://mlsanity.com/paper/1905.09788/) (2019)
- [Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models](https://mlsanity.com/paper/2501.06751/) (2025)
- [Self-Attentive Pooling for Efficient Deep Learning](https://mlsanity.com/paper/2209.07659/) (2022)
- [Enhancing Sentence Embedding with Generalized Pooling](https://mlsanity.com/paper/1806.09828/) (2018)

## Abstract (arXiv)

This paper presents system description for Arabic medical text classification across 82 distinct categories. Our primary architecture utilizes a fine-tuned AraBERTv2 encoder enhanced with a hybrid pooling strategies, combining attention and mean representations, and multi-sample dropout for robust regularization. We systematically benchmark this approach against a suite of multilingual and Arabic-specific encoders, as well as several large-scale causal decoders, including zero-shot re-ranking via Llama 3.3 70B and feature extraction from Qwen 3B hidden states. Our findings demonstrate that specialized bidirectional encoders significantly outperform causal decoders in capturing the precise semantic boundaries required for fine-grained medical text classification. We show that causal decoders, optimized for next-token prediction, produce sequence-biased embeddings that are less effective for categorization compared to the global context captured by bidirectional attention. Despite significant class imbalance and label noise identified within the training data, our results highlight the superior semantic compression of fine-tuned encoders for specialized Arabic NLP tasks. Final performance metrics on the test set, including Accuracy and Macro-F1, are reported and discussed.