Hybrid Particle-Voxel Dynamics
for Multimodal 3D World Simulation

2Multimodal 3D Video Prediction

3D World Simulator supports diverse robot manipulation tasks, spanning deformable objects, articulated objects and in-hand manipulation. Pick a task below to see an open-loop rollout of the model: it is given a few real frames to start from and then runs on its own from the recorded actions. Because the prediction is 3D, the same rollout can be viewed in several ways: rendered from a novel viewpoint, as the predicted 3D particles with their tactile signal, as the intermediate voxel representation, and as the latent representation the dynamics model operates on.

3Interactive demo

The model runs on a GPU and streams its render to the browser. Press start and a container is started for you, which takes about 45 seconds the first time. Then move the arms with the keys below, or drag the handles that appear on the grippers. It is a research demo rather than a product: if you push the arms somewhere the training data never went, the cloth will look strange.

4Unified composition of simulation "backbones"

5Controllable Variation in Backgrounds and Robot Embodiments

The 3D nature of our model makes it far more controllable and editable than 2D video prediction models. The base model does not rely on segmentation masks, but it can be fine-tuned to predict the manipulated scene on top of a separately captured background, so the prediction and the room become independent assets. This lets us place a rollout in different backgrounds and draw it with different robot embodiments, posed from the model's predicted joint states. And since the output is 3D Gaussians, existing relighting methods apply directly. Below, one box episode is carried through four scanned rooms, the first with the original arms and the rest with a pair of SO-101 arms, ending with four tasks side by side.

6Policy evaluation in the model

A policy can be run inside the model instead of on the robot: we render the camera views the policy expects, it returns an action, and the model predicts what happens next. The model's prediction is detailed enough to tell a success from a failure. For each task we picked one rollout that ends in success and one that does not. For Push-T the goal pose is drawn as an outline; for the cloth we look at whether the towel ends up folded. Plug insertion compares two different policies from the same start.

7Policies trained purely in the model

This one is real footage, not a render. The policies here were trained only on data produced by the world model and are running on the robot, sped up 20×.

8Scaling to diverse objects

Deform360 is the same kind of model trained on a collection of deformable objects (hats, plush toys, towels, ropes, sleeves). The camera starts on one object and pulls back to show the rest. All of them are predictions.

9How it works

Hybrid Particle-Voxel Autoencoder

The input to the model consists of a colored point cloud obtained from several well-placed stereo cameras that cover desired scene viewpoints. In our setup, we use three static cameras and two wrist-mounted cameras.

The encoder first voxelizes the input and downsamples voxels into a compact voxel latent representation with sparse 3D convolutions. The decoder then reconstructs the input point cloud by first upsampling the latent into finer voxels, then using each occupied voxel as a seed to upsample particles in free space. Each particle parameterizes a 3D Gaussian, enabling rendering from arbitrary viewpoints. Additionally, tactile signals can be represented as 3D particles, enabling spatially aligned generation.

3D Latent Dynamics Model

After training the hybrid particle-voxel autoencoder, we then train an action-conditioned dynamics model over its 3D latent representations. Given a history of latents and an input action, it predicts the next latent representation. The dynamics model can then generate an open loop trajectory via autoregressive prediction.

We instantiate our dynamics model as a 3D U-Net. It downsamples the voxel latents further using 3D convolutions, then applies interleaved within-frame spatial attention and across-frame temporal attention. We use direct next-latent prediction and do not rely on any diffusion or flow matching approaches.

Architecture Motivation

We use a hybrid particle and voxel structure to handle the generation of particles in free space while creating a predictable latent representation for the dynamics model.

Point clouds can contain arbitrary numbers of particles at arbitrary spatial locations. Without explicitly maintaining persistent particle identities, point clouds are inherently unordered and change cardinality between timesteps, making it hard to find a consistent target that a dynamics model can predict frame-to-frame. As a result, we use a voxel latent representation that provides fixed spatial locations which provide consistent, predictable targets. Since voxels can be too coarse and expensive, we upsample particles from upsampled voxels to try to capture the best of both worlds.

Most 3D dynamics models circumvent this issue by predicting particle velocities through maintaining explicit particle identities and estimating correspondences. We intentionally do not take this route as we include tasks with deformable objects and in-hand manipulation, where pose estimation or estimating particle correspondences can be difficult under substantial occlusions.

In this work, we make the assumption that the complex interactions of interest happen within a manually set workspace, and that any scene components outside this workspace are static or predictable with analytic models. To model the background outside of the workspace, we utilize a 3D Gaussian Splat fit to a room scan. To model the robot arms, we use a 3D Gaussian Splat augmented with a kinematic model that can

Because the room and the robot are separate assets, both can be swapped without touching the prediction. Below, one box frame is carried through four scanned rooms, and the dining room is then drawn with two other robot embodiments.