Modeling Diverse Treatment Policies from Observational Health Data

Abstract

Learning policies for real world tasks often requires modeling human behavior, especially in domains like healthcare and driving. In these settings, skills are learned from expert human demonstrations, but such data are typically multimodal, violating the common single expert assumption. We study sequential clinical treatment decision making in the offline imitation learning setting, where environment interaction is prohibited, reflecting the challenges of experimentation in safety critical domains. Existing methods for multi expert offline imitation learning often restrict the latent space, underspecify its structure, or omit objective terms that prevent latent collapse and encourage behavior discovery. We propose a fully offline approach that addresses these shortcomings and improves learning from multi expert demonstrations through modifications to the formulation of the latent approximate posterior and the model architecture. We suggest that our method is more robust to real world settings where the true number of demonstrators may not be known. We also incorporate an occupancy matching term into our objective that injects awareness of the rollout distribution over trajectories into our behavior cloning objective. We evaluate our method against baselines on both simulated multi expert demonstrations from an extended S-CVSim and real world demonstrations from MIMIC. Our approach achieves consistently higher next step action prediction and behavior discovery performance. While ground truth expert policies are unavailable for MIMIC, visual analysis shows our method uncovers clinically meaningful variations in expert strategies, reflecting treatment population diversity.