KineState-SWAM

A Surgical World Action Model with Explicit Kinematics and Causal Interaction Dynamics

Abstract

KineState-SWAM links surgical robot action prediction and video generation through a persistent interaction state. Calibrated URDF kinematics convert predicted joint trajectories into tool motion and instrument renderings. The model uses these signals to update its interaction state and condition future video frames. The paper also introduces dVRK-Chole5.2k, a dataset of 5,255 videos spanning 17 cholecystectomy phases.

Comparison of four world-modeling approaches with KineState-SWAM training and recurrent inference
KineState-SWAM uses URDF-derived tool motion to update the state shared by action prediction and video generation.

Method

KineState-SWAM architecture showing the flow-matching action policy, kinematic geometry conditioning, causal interaction dynamics, and the State–Geometry Adapter connecting to the video generator
The action policy predicts joint trajectories. Causal Interaction Dynamics updates the state from tool motion, and the State–Geometry Adapter feeds both the state and projected geometry to the video generator.

Video demo

Clip-placement rollout using an edited PSM1 trajectory. The views show generated RGB, URDF geometry with the tool-tip path, and their overlay.