A simulator with no simulation in it
Nvidia has released Cosmos3-H-Dreams, a model that generates surgical video in real time in response to instrument controls. Move the grippers and the image of the operating field changes to match.
What makes it unusual is what is missing. A conventional surgical simulator is a physics engine: tissue is modelled, tools collide with it, and a renderer draws the result. Cosmos3-H-Dreams has none of that. It is a generative video model that was trained on recordings of real operations and predicts the next frames from the current frames and the surgeon’s inputs.
The training data is clinical cholecystectomy footage — gallbladder removals — recorded on CMR Surgical’s Versius robotic platform. The action space is 28 dimensions covering pose and gripper data for both arms, energy controls, and thumbstick axes and buttons, aligned with video at 10 Hz.
What was published
Two checkpoints went out on 20 October: a four-step student and a two-step student. The two-step model was trained separately rather than produced by cutting the four-step model’s sampling steps, which is the detail that tells you the shorter model is a real training run and not a quality compromise applied after the fact.
The base is Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment. Video is 832 by 480. The teacher was trained on 24-frame sequences and then extended to 72 frames, which is 7.2 seconds at 10 Hz. Nvidia says it runs interactively on a single RTX PRO 6000 GPU, using a streaming runtime called FlashDreams with an adapted memory module that consolidates the key-value cache during long rollouts.
Alongside the checkpoints, Nvidia published code and examples on GitHub and a recipe for teacher fine-tuning and distillation, so a hospital or a device maker can adapt the system to its own recordings.
The caveat is in the post
Nvidia describes it as a research simulator and states that it is not a validated system for patient care. That line is doing real work. A model that produces plausible surgical video from a 16-billion-parameter video prior has no guarantee that the tissue behaves as tissue does — only that the frames look like the frames it was trained on.
Where it came from and what is unstated
The predecessor, Cosmos-H-Dreams, was built on Cosmos-Predict2.5 and targeted tabletop suturing on the dVRK research platform. Moving from a research rig doing suturing drills to clinical recordings of a commercial surgical robot doing a real procedure is the step this release represents.
Two things the post does not give: the parameter count of the distilled checkpoints themselves, and the frame rates and latencies actually achieved at runtime. For a model whose claim is that it is interactive, the second omission is the one to press on. The checkpoint is on Hugging Face as nvidia/Cosmos3-H-Dreams, tagged image-to-video, and the licence terms are on the model card rather than in the announcement.
