HounsWorld: A Multimodal World Model for Hidden Patient-State Readout, Reconstruction, and Simulation
On the map: World models → Medical and surgical world models
Abstract (arXiv)
Clinical intelligence requires estimating a patient's underlying condition from incomplete observations rather than learning isolated mappings from scans to answers. Volumetric medical images provide dense observations of anatomy, attenuation, and lesions, whereas clinical language provides sparse but complementary semantic observations. We formulate CT-centered intelligence as inference over a shared latent patient state, under which readout, reconstruction, and simulation all become state-dependent prediction problems. To operationalize this view, we introduce HounsBench, a computed tomography (CT) centric patient-state benchmark that unifies these three task families with patient-disjoint splits and per-family metrics, and HounsWorld, a 3B multimodal world model that treats volumetric scans and language as observations of the shared state through Joint Understanding-Generation Learning. A shared transformer forms an implicit patient-state estimate and supports three outputs: query-conditioned answers that read out the state, reports and captions that reconstruct it in language, and condition-specific CT volumes for low-dose denoising, virtual contrast enhancement, and anatomy-constrained text-and-mask-to-volume generation. Zero-initialized CT adapters preserve pretrained multimodal mappings, while condition-explicit Hounsfield-unit window sampling exposes clinically meaningful density observations. HounsWorld shows strong performance across all three task families while consistently improving CT understanding through clinically structured completion. Our project is available at https://github.com/byhwhite/HounsWorld.git
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and surgical world models, one of 11 directions in World models. Builds on 43 of its 52 references that are in the map. First posted on arXiv 13 Aug 2026.
Builds on
43 of 52 references.
Closest ancestors · 2
Same direction, newest first
- 2025Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planning2 Jun
- 2025CheXWorld: Exploring Image World Modeling for Radiograph Representation Learning18 Apr
Other references · 41
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
- 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
- 2018World Modelscited by 1,513
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
29 more
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2024Genie: Generative Interactive Environmentscited by 747
- 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
- 2022TotalSegmentator: robust segmentation of 104 anatomical structures in CT imagescited by 613
- 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
- 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
- 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
- 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 200
- 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 199
- 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
- 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 162
- 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 114
- 2023Diffusion Model as Representation Learnercited by 95
- 2024MAISI: Medical AI for Synthetic Imagingcited by 87
- 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
- 2023GenerateCT: Text-Conditional Generation of 3D Chest CT Volumescited by 64
- 2023Masked Image Training for Generalizable Deep Image Denoisingcited by 57
- 2023PRIOR: Prototype Representation Joint Learning from Medical Images and Reportscited by 55
- 2024RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysiscited by 42
- 20253D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Taskscited by 26
- 2021CyTran: A Cycle-Consistent Transformer with Multi-Level Consistency for Non-Contrast to Contrast CT Translationcited by 15
- 2024Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Modelcited by 15
- 2025Unified Multimodal Models as Auto-Encoderscited by 13
- 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 12
- 2026OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysiscited by 10
- 2024Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Modelscited by 5
- 2025See More, Change Less: Anatomy-Aware Diffusion for Contrast Enhancementcited by 3
- 2026A World Model of Radiologist Reading for Medical Image Representation Learningcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Tasks
- 2026Learning How Much, Not Just What: Cross-Patient Burden Order for CT Vision-Language Pretraining
- 2026A radiographic world model for clinical reasoning and evidence generation
- 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
- 20263DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis
- 2025MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine
- 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis
- 2026Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT