Modeling Dynamic Objects in Scenes with Generative Particle Systems

Abstract

Humans readily interpret the motion of deformable and rigid bodies, even when encountering unfamiliar objects with minimal shape or texture cues. In such cases, motion serves as a critical signal for recognition and understanding. Inspired by this ability, we propose a generative model that represents 3D matter as small Gaussians (“particles”) drawn from clusters capturing groups of coherently moving matter. We develop an e!cient inference algorithm based on parallelized block Gibbs sampling to recover stable particle motion and rigid groupings. Our model provides a tractable, object-centric generalization of as-rigidas-possible (ARAP) regularizers used in motion tracking. To assess alignment with human perceptual judgments, we test our approach on random dot kinematograms—sparse motion displays in which dot trajectories convey latent object structure, often used to probe visual understanding of motion and grouping. In this setting, our approach captures human-like responses, including graded patterns of uncertainty across ambiguous conditions. Applied to naturalistic RGB videos, it infers dense particle representations that track object motion and deformation over time. These results demonstrate that our model enables persistent latent scene structure suitable for object-level reasoning.