NVIDIA Releases Four New Open-Weight Cosmos 3 Models for Interactive World Simulation

NVIDIA has expanded its Cosmos 3 world model development platform by introducing four new open-weight checkpoints. These models, part of the Cosmos 3 Nano family, are designed for generating interactive world simulations and multi-sensor multi-view driving scenes. Each model has 16 billion trainable parameters and is available under the OpenMDW1.1 license, permitting both commercial and non-commercial use. This expansion aims to provide researchers and developers with tools to create richer, more responsive simulated worlds.

Cosmos3-Nano-Transfer-Auto: Multi-View Driving Data Generation

Cosmos3-Nano-Transfer-Auto is a bidirectional model for offline synthetic driving data generation. It generates synchronized video across up to 11 camera views, including fisheyes, and supports joint RGB–LiDAR generation. Per-camera World Scenario Map (WSM) controls specify scene structure and traffic motion, while text prompts guide appearance. Developers can vary environmental conditions to expand training and evaluation coverage, with optional initial RGB images to anchor scene appearance. Joint LiDAR generation additionally uses aligned HD-map range-map controls. Transfer-Auto also serves as the foundation for adapting Cosmos to your own sensor rig and driving domain by post-training it on your data.

Cosmos3-Nano-Sim-Auto: Streaming Driving Observations

Cosmos3-Nano-Sim-Auto is the causally distilled streaming variant of Transfer-Auto. It generates successive small chunks of synchronized camera video and LiDAR in just a few denoising steps, using generated history and updated driving controls to support fast streaming and long, interactive rollouts. Provide per-camera captions and synchronized WSM controls, optionally adding initial RGB images. For LiDAR generation, include HD-Map range controls aligned with the camera controls. The distilled model generates successive chunks using its rolling history. Closed-loop interaction requires a simulation runtime and WSM renderer to update those controls from driving inputs.

Cosmos3-Nano-Sim-Transfer: Simulator Controls to Photorealistic Video

Cosmos3-Nano-Sim-Transfer uses a physics simulator's spatial controls and a text prompt to render color observations that a policy can consume. The simulator computes motion, contacts, and physical state. Sim-Transfer uses its spatial controls and a text prompt to render color observations. Depth (relative distance from the camera) and segmentation (regions and object boundaries) are supported. Text guides appearance; physics and ground-truth state stay with the simulator.

Cosmos3-Nano-Sim-Bimanual: Robot Action Video Prediction

Cosmos3-Nano-Sim-Bimanual simulates how an environment evolves in response to a bimanual robot's actions. It supports interactive two-arm simulation for YAM and AgiBot embodiments, controlled by a policy or human-operated joysticks. Start with an image, move the robot or camera, and generate the next observations. One control sequence can switch between viewpoint changes and manipulation. The mixed-action example uses 59 control values, padded to 64 during training. Its schema defines coordinate frames, rotations, gripper calibration, validity masks, and control domains.

Applications in Robotics and Autonomous Driving

For robotics, Sim-Bimanual predicts visual consequences from actions, while Sim-Transfer turns a physics simulator's controls into generated observations. For autonomous driving, synchronized camera and LiDAR generation connects visual observations with 3D sensing. Scene and traffic-signal controls let researchers vary road layouts and signal states, then study how a driving policy responds. Streaming generation connects video prediction to ongoing interaction, giving world-model researchers a way to explore how actions and observations unfold together.

Hardware Requirements and Availability

The models require a Linux system, an NVIDIA GPU, Docker, and the NVIDIA Container Toolkit. The published vLLM-Omni setup was tested on one GB200. NVIDIA's vision is to help researchers and developers create richer, more responsive simulated worlds. Open weights and post-training recipes provide a shared starting point for adapting models, comparing methods, and contributing improvements.