# Proof-of-TBI -- Fine-Tuned Vision Language Model Consortium and OpenAI-o3 Reasoning LLM-Based Medical Diagnosis Support System for Mild Traumatic Brain Injury (TBI) Prediction

arXiv: https://arxiv.org/abs/2504.18671 · HTML: https://mlsanity.com/paper/2504.18671/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)
- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 12 of 12)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Towards Accurate Differential Diagnosis with Large Language Models](https://mlsanity.com/paper/2312.00164/) (2023)
- [LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models](https://mlsanity.com/paper/2404.01230/) (2024)
- [SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification](https://mlsanity.com/paper/2211.16191/) (2022)
- [DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledge](https://mlsanity.com/paper/2405.12541/) (2024)
- [ThoughtSource: A central hub for large language model reasoning data](https://mlsanity.com/paper/2301.11596/) (2023)
- [Large Language Models for Interpretable Mental Health Diagnosis](https://mlsanity.com/paper/2501.07653/) (2025)

## Built on by (most cited first, 11 of 11)

- [A Practical Guide for Designing, Developing, and Deploying Production-Grade Agentic AI Workflows](https://mlsanity.com/paper/2512.08769/) (2025)
- [Towards Responsible and Explainable AI Agents with Consensus-Driven Reasoning](https://mlsanity.com/paper/2512.21699/) (2025)
- [Standardization of Psychiatric Diagnoses -- Role of Fine-tuned LLM Consortium and OpenAI-gpt-oss Reasoning LLM Enabled Decision Support System](https://mlsanity.com/paper/2510.25588/) (2025)
- [ASTRIDE: A Security Threat Modeling Platform for Agentic-AI Applications](https://mlsanity.com/paper/2512.04785/) (2025)
- [Standardization of Neuromuscular Reflex Analysis -- Role of Fine-Tuned Vision-Language Model Consortium and OpenAI gpt-oss Reasoning LLM Enabled Decision Support System](https://mlsanity.com/paper/2508.12473/) (2025)
- [Flowr -- Scaling Up Retail Supply Chain Operations Through Agentic AI in Large Scale Supermarket Chains](https://mlsanity.com/paper/2604.05987/) (2026)
- [Toward Zero-Egress Psychiatric AI: On-Device LLM Deployment for Privacy-Preserving Mental Health Decision Support](https://mlsanity.com/paper/2604.18302/) (2026)
- [Think Before You Act -- A Neurocognitive Governance Model for Autonomous AI Agents](https://mlsanity.com/paper/2604.25684/) (2026)
- [MindGap: A Conversational AI Framework for Upstream Neuroplastic Intervention in Post-Traumatic Stress Disorder](https://mlsanity.com/paper/2605.14660/) (2026)
- [Train the Trainers -- An Agentic AI Framework for Peer-Based Mental Health Support in Battlefield Environments](https://mlsanity.com/paper/2605.16269/) (2026)
- [Human-AI Agent Interaction as a Neuroplastic Training Environment](https://mlsanity.com/paper/2607.12823/) (2026)

## Abstract (arXiv)

Mild Traumatic Brain Injury (TBI) detection presents significant challenges due to the subtle and often ambiguous presentation of symptoms in medical imaging, making accurate diagnosis a complex task. To address these challenges, we propose Proof-of-TBI, a medical diagnosis support system that integrates multiple fine-tuned vision-language models with the OpenAI-o3 reasoning large language model (LLM). Our approach fine-tunes multiple vision-language models using a labeled dataset of TBI MRI scans, training them to diagnose TBI symptoms effectively. The predictions from these models are aggregated through a consensus-based decision-making process. The system evaluates the predictions from all fine-tuned vision language models using the OpenAI-o3 reasoning LLM, a model that has demonstrated remarkable reasoning performance, to produce the most accurate final diagnosis. The LLM Agents orchestrates interactions between the vision-language models and the reasoning LLM, managing the final decision-making process with transparency, reliability, and automation. This end-to-end decision-making workflow combines the vision-language model consortium with the OpenAI-o3 reasoning LLM, enabled by custom prompt engineering by the LLM agents. The prototype for the proposed platform was developed in collaboration with the U.S. Army Medical Research team in Newport News, Virginia, incorporating five fine-tuned vision-language models. The results demonstrate the transformative potential of combining fine-tuned vision-language model inputs with the OpenAI-o3 reasoning LLM to create a robust, secure, and highly accurate diagnostic system for mild TBI prediction. To the best of our knowledge, this research represents the first application of fine-tuned vision-language models integrated with a reasoning LLM for TBI prediction tasks.