# Bag of Tricks or Bag of Myths? Reducing Modeling Complexity with Task Knowledge in Explainable Suicide Risk Assessment

arXiv: https://arxiv.org/abs/2609.07766 · HTML: https://mlsanity.com/paper/2609.07766/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 57 references that are in the map.
- First posted on arXiv 7 Sep 2026.

## Builds on (most cited first, 30 of 33)

- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [What is the Effect of Importance Weighting in Deep Learning?](https://mlsanity.com/paper/1812.03372/) (2018)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [With Little Power Comes Great Responsibility](https://mlsanity.com/paper/2010.06595/) (2020)
- [On the State of the Art of Evaluation in Neural Language Models](https://mlsanity.com/paper/1707.05589/) (2017)
- [Self-training Improves Pre-training for Natural Language Understanding](https://mlsanity.com/paper/2010.02194/) (2020)
- [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://mlsanity.com/paper/1707.09861/) (2017)
- [Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities](https://mlsanity.com/paper/2111.08851/) (2021)
- [How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?](https://mlsanity.com/paper/2010.01764/) (2020)
- [Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study](https://mlsanity.com/paper/2103.06511/) (2021)
- [Knowledge-intensive Language Understanding for Explainable AI](https://mlsanity.com/paper/2108.01174/) (2021)
- [Process Knowledge-Infused AI: Towards User-level Explainability, Interpretability, and Safety](https://mlsanity.com/paper/2206.13349/) (2022)
- [FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding](https://mlsanity.com/paper/2109.12742/) (2021)
- [Characterization of Time-variant and Time-invariant Assessment of Suicidality on Reddit using C-SSRS](https://mlsanity.com/paper/2104.04140/) (2021)
- [A Systematic Evaluation of LLM Strategies for Mental Health Text Analysis: Fine-tuning vs. Prompt Engineering vs. RAG](https://mlsanity.com/paper/2503.24307/) (2025)
- [Building Trustworthy NeuroSymbolic AI Systems: Consistency, Reliability, Explainability, and Safety](https://mlsanity.com/paper/2312.06798/) (2023)
- [WellDunn: On the Robustness and Explainability of Language Models and Large Language Models in Identifying Wellness Dimensions](https://mlsanity.com/paper/2406.12058/) (2024)
- [Leveraging Large Language Models for Suicide Detection on Social Media with Limited Labels](https://mlsanity.com/paper/2410.04501/) (2024)
- [Process Knowledge-infused Learning for Clinician-friendly Explanations](https://mlsanity.com/paper/2306.09824/) (2023)

## Abstract (arXiv)

Assessing suicide risk from social media text is a small-data, high-stakes setting requiring not only severity prediction but also supporting evidence and clinically relevant risk and protective factors. Yet common NLP techniques, including model scaling, synthetic data, loss reweighting, ensembling, and threshold tuning, are often applied without testing whether their gains hold up under severe class imbalance, coupled outputs, and limited author-level data. We study 1,635 clinician-annotated posts and audit 31 pre-specified techniques from 7 methodological families through roughly 300 controlled experiments on author-disjoint partitions. We found no prior audit of this playbook in this regime. The findings guide a task-grounded system for three outputs: 4-level suicide risk, evidence spans, and 24 clinical risk and protective factors. Only 5 of 31 comparisons produced reliable gains. We reformulate factor prediction as entailment between each post and its codebook definitions, using an architecturally diverse ensemble with class-balanced training and score rescaling. Risk predictions condition a 7-model evidence tagger ensemble; evidence restricts symbolic risk rules; and a difficult risk class is routed separately. The factor predictor remains independent because risk evidence provides no additional factor signal. We also correct a mismatch between validation scores used for threshold fitting and test-time ensemble scores through deployment-consistent calibration, yielding the largest improvement to the factor system. The final system achieves 0.8203 for risk, 0.7953 for evidence, and 0.7045 macro-F1 for factors, with a 0.7781 composite, ranking third among 53 teams. We call the underlying principle task-conditioned technique selection: retain techniques only when task-specific knowledge, structure, or empirical evidence justifies them.