mlsanityData to 14 Oct 2026

HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulation

arXiv 2608.12904 ↗First posted 13 Aug 2026cs.CVcited by 0

On the map: World models → Medical and surgical world models

Abstract (arXiv)

Clinical intelligence requires estimating a patient's underlying condition from incomplete observations rather than learning isolated mappings from scans to answers. Volumetric medical images provide dense observations of anatomy, attenuation, and lesions, whereas clinical language provides sparse but complementary semantic observations. We formulate CT-centered intelligence as inference over a shared latent patient state, under which readout, reconstruction, and simulation all become state-dependent prediction problems. To operationalize this view, we introduce HounsBench, a computed tomography (CT) centric patient-state benchmark that unifies these three task families with patient-disjoint splits and per-family metrics, and HounsWorld, a 3B multimodal world model that treats volumetric scans and language as observations of the shared state through Joint Understanding-Generation Learning. A shared transformer forms an implicit patient-state estimate and supports three outputs: query-conditioned answers that read out the state, reports and captions that reconstruct it in language, and condition-specific CT volumes for low-dose denoising, virtual contrast enhancement, and anatomy-constrained text-and-mask-to-volume generation. Zero-initialized CT adapters preserve pretrained multimodal mappings, while condition-explicit Hounsfield-unit window sampling exposes clinically meaningful density observations. HounsWorld shows strong performance across all three task families while consistently improving CT understanding through clinically structured completion. Our project is available at https://github.com/byhwhite/HounsWorld.git

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and surgical world models, one of 11 directions in World models. Builds on 43 of its 52 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

43 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planning2 Jun
  2. 2025CheXWorld: Exploring Image World Modeling for Radiograph Representation Learning18 Apr

Other references · 41

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2025Qwen3-VL Technical Reportcited by 2,793
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  8. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  10. 2018World Modelscited by 1,513
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  12. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
29 more
  1. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  2. 2024Genie: Generative Interactive Environmentscited by 747
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  4. 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 613
  5. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  6. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  7. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  8. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  9. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 200
  10. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 199
  11. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
  12. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 162
  13. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 114
  14. 2023Diffusion Model as Representation Learnercited by 95
  15. 2024MAISI: Medical AI for Synthetic Imagingcited by 87
  16. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  17. 2023GenerateCT: Text-Conditional Generation of 3D Chest CT Volumescited by 64
  18. 2023Masked Image Training for Generalizable Deep Image Denoisingcited by 57
  19. 2023PRIOR: Prototype Representation Joint Learning from Medical Images and Reportscited by 55
  20. 2024RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysiscited by 42
  21. 20253D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Taskscited by 26
  22. 2021CyTran: A Cycle-Consistent Transformer with Multi-Level Consistency for Non-Contrast to Contrast CT Translationcited by 15
  23. 2024Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Modelcited by 15
  24. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  25. 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 12
  26. 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 10
  27. 2024Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Modelscited by 5
  28. 2025See More, Change Less: Anatomy-Aware Diffusion for Contrast Enhancementcited by 3
  29. 2026A World Model of Radiologist Reading for Medical Image Representation Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Tasks
  2. 2026Learning How Much, Not Just What: Cross-Patient Burden Order for CT Vision-Language Pretraining
  3. 2026A radiographic world model for clinical reasoning and evidence generation
  4. 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
  5. 20263DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis
  6. 2025MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine
  7. 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis
  8. 2026Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT