mlsanityData to 7 Oct 2026

A global log for medical AI

arXiv 2510.04033 ↗First posted 5 Oct 2025cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern computer systems rely on syslog, a universal protocol that records critical events across heterogeneous infrastructure. Medicine's rapidly growing AI stack has no equivalent. As medicine deploys AI tools at scale, there is no standard way to record how, when, by whom, and for whom these models are used. Without such records, it is difficult to measure real-world performance and outcomes, detect adverse events, or identify bias and dataset drift. Here we introduce MedLog, a protocol for event-level logging of medical AI. Each time an AI model interacts with a human, another algorithm, or an automated workflow, MedLog creates a record. Each record contains nine core fields: header, model, user, target, inputs, artifacts, outputs, outcomes, and feedback. We apply MedLog across four deployments in the US, Switzerland, and Vietnam: ICU deterioration prediction, tetanus progression monitoring from wearable signals, automated sepsis quality reporting, and patient attendance prediction. MedLog records capture model behavior, workflow interactions, and downstream outcomes, including AI performance degradation during severe weather events in patient attendance prediction and increased laboratory testing after ICU deterioration alerts. MedLog limits the data footprint through risk-based sampling, lifecycle-aware retention policies, and write-behind caching, enabling deployment in low-resource settings. It also supports detailed traces for complex, agentic, or multi-stage workflows, creating a foundation for continuous monitoring, auditing, and improvement of medical AI.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 123 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

48 of 123 references.

References · 48

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2018Model Cards for Model Reportingcited by 1,068
  8. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  9. 2020The Future of Digital Health with Federated Learningcited by 625
  10. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  12. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
36 more
  1. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
  2. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  3. 2023Towards Generalist Biomedical AIcited by 286
  4. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  5. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  6. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  7. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  8. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 155
  9. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  10. 2019A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysiscited by 150
  11. 2021Disparities in Dermatology AI: Assessments Using Diverse Clinical Imagescited by 132
  12. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  13. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  14. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  15. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 100
  16. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  17. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 74
  18. 2020Loghub: A Large Collection of System Log Datasets for AI-driven Log Analyticscited by 68
  19. 2022CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarkscited by 68
  20. 2025MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Modelscited by 68
  21. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 60
  22. 2023FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcarecited by 59
  23. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  24. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  25. 2022Diagnosing failures of fairness transfer across distribution shift in real-world medical settingscited by 45
  26. 2024Extracting Prompts by Inverting LLM Outputscited by 40
  27. 2024Prompt Stealing Attacks Against Large Language Modelscited by 39
  28. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  29. 2023LogGPT: Exploring ChatGPT for Log-Based Anomaly Detectioncited by 33
  30. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  31. 2024Superhuman performance of a large language model on the reasoning tasks of a physiciancited by 28
  32. 2023Benchmarking Large Language Models for Log Analysis, Security, and Interpretationcited by 19
  33. 2025Improving Medical Reasoning with Curriculum-Aware Reinforcement Learningcited by 16
  34. 2022System Log Parsing: A Surveycited by 10
  35. 2025AI-based Clinical Decision Support for Primary Care: A Real-World Studycited by 10
  36. 2025A Systematic Survey of Model Extraction Attacks and Defenses: State-of-the-Art and Perspectivescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
  2. 2026MedCTA: A Benchmark for Clinical Tool Agents
  3. 2025Auditing medical multi-agent AI reveals risks of false consensus
  4. 2025Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences
  5. 2025MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Book
  6. 2024Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
  7. 2026ART: Action-based Reasoning Task Benchmarking for Medical AI Agents
  8. 2026Vital Trace: Protocol-Constrained Patient-State Reasoning for Longitudinal Clinical Trajectories