Google DeepMind's Genie 4 World Model Adds Audio, Longer Scenes, and Real-Time Editing

Google DeepMind released Genie 4, its newest world model, on Tuesday. The model generates photorealistic and interactive worlds with real-time audio, allowing users to edit scenes mid-simulation. Genie 4 can maintain consistent scenes for up to two minutes, double the limit of its predecessor Genie 3, and offers better physics and more humanlike characters.

What Genie 4 Actually Does

Genie 4 is Google's latest world model, designed to create immersive, interactive environments. Unlike previous versions, it includes real-time audio generation, such as music, ambient noises, and interactive dialogue. Users can type prompts mid-simulation to change the scene, adjusting elements like weather, camera angles, or adding new objects and people. The model is also more closely integrated with Gemini, Google's AI model, which acts as a director to plan steps and check frames while Genie creates the visuals.

In a research experiment tied to Gemini Robotics, a Gemini 4 Argon model broke down tasks into steps that Genie 4 simulated in real time. This experiment shows a possible way to test robot scenarios in simulation before they happen in the real world. Tejas Kulkarni, research director at Google, said during a briefing: "What we'll increasingly see in the future are these hybrid world models, where you have a combination of Gemini writing code and text, and Genie ingesting that in real time and simulating reality at different levels of abstraction."

How Genie 4 Compares to Previous Models and Competitors

Genie 4 builds on Genie 3, which was released as a research preview over a year ago. The new model offers significant improvements, including longer scene consistency, better physics, and more realistic characters. It also includes real-time audio, a feature not present in Genie 3.

The release of Genie 4 comes a couple of months after AI video startup Runway released its second world model, GWM Worlds 2, a playable model that responds dynamically to user inputs and includes audio generation. The world model industry is beginning to consolidate, with Fei-Fei Li's World Labs being acquired by AMD, Amazon in talks to buy Decart, and Rivian acquiring Third Dimension to bring simulation capabilities in-house for self-driving testing.

Potential Applications: Gaming, Filmmaking, and Robotics

The advancements in Genie 4 have potential applications in various fields. For gamers, it could mean more immersive and interactive gaming experiences. Filmmakers could use it for prototyping and storyboarding, allowing for rapid ideation and scene planning. Diego Rivas, a Google DeepMind engineer, said during a press briefing: "The real time aspect becomes most valuable in the areas where keeping a state of flow in the creativity space becomes incredibly useful. For example, in the ideation phase, in the storyboarding phase — anytime you're kind of in need of rapid ideation."

In robotics, Genie 4 could be used to simulate and test scenarios before they are implemented in the real world. One speculative example of interactive learning would be entering a painting of a historical scene and chatting to the painting's subjects about the event portrayed.

Rivas added that users could eventually rely on Gemini Omni, a multimodal large language model that can generate video from images, audio, and text, to turn those prototypes into creative assets, though those capabilities are still a ways off.

Current Availability and Limitations

Currently, Genie 4 is available as a research preview for trusted testers. In the coming weeks, it will roll out to Google AI Ultra subscribers in Project Genie and early access partners of the Gemini API and Google AI Studio.

Throughout the briefing, Rivas stressed that it's still early days for Genie 4 and that it's not yet clear what the end-to-end product experience will be. The team acknowledged that Genie 4 still has gaps: details and objects can drift the longer a simulation runs, and they haven't quite figured out how to ensure the audio will be accurate and any in-world text will be legible.