Are World Models That Enable Real-Time Interaction the Optimal Choice?

A year ago, Google introduced Genie 3, a ‘world model capable of real-time interaction.’ Now, Genie 4, hailed as the ‘most realistic and interactive world model to date,’ has made its debut.

Genie has never been content with merely ‘generating realistic videos.’ Its ambition is to create a virtual world that operates in real-time before your eyes—a world that can listen, respond, and adapt.

It is crucial to note that this release is still a ‘research preview’ and is currently accessible only to invited testers. Eventually, it will be made available to Ultra subscribers, with regular users having to await their turn. Leitech (ID: Leitech) has already submitted an application and will share hands-on experiences once access is granted.

Over the past two years, companies in autonomous driving, robotics, and video generation have focused on the concept of ‘world models.’ The distinction lies in the fact that in autonomous driving and robotics, world models are employed to generate a virtual environment for model training and to predict real-world events in real-time. Google’s Genie 4, however, can not only generate a virtual world but also modify its state based on user commands.

Real-Time Interaction: The New Frontier for World Models

According to Google, Genie 4 inherently supports multimodality, synchronously generating music, ambient sound effects, interactive dialogue, and virtual worlds.

To ensure performance in long-term scenarios, Genie 4 is integrated and expanded with the Gemini model. Gemini Omni anchors the behavioral logic of simulated scenes to the real world, while other models handle the complex reasoning required to maintain simulation coherence. This results in more realistic rendering of sounds like waves crashing on the beach or leaves rustling in the wind.

(Video source: Google; image produced by Leitech)

When Genie 4 generates a world in real-time, Gemini continuously monitors the scene, breaks down goals into a series of steps, and reviews received frames to track progress and verify the success of each step. Leveraging Genie 4’s understanding of the physical world, changes in object shapes and trajectories caused by airflow and collisions are realistically rendered in the generated videos.

Most importantly, users can modify the environment, introduce new objects or characters, and observe how existing characters react to these changes and the environment during video generation. This interactive capability is lacking in some world models.

From this perspective, Genie 4’s most apparent application seems to be in video generation.

After Sora sparked interest in the video generation field (translation note: ‘field’ refers to the competitive arena or domain), domestic companies like Kuaishou and ByteDance joined the fray, with AI models like Kling and Jiemeng continuously improving in capability and generating increasingly realistic visuals. Unlike these models, Genie 4 prioritizes not just visual effects but also real-time interactivity.

(Image source: Screenshot of Jiemeng)

When using ordinary video generation AI, we typically input instructions, upload images or video clips, and then the model generates a video based on our input. During generation, we cannot intervene; if the video deviates from our plan, we must regenerate it or make post-production adjustments.

When Kling and Jiemeng took the industry by storm, Leitech stated that for ordinary people, the greatest value of video generation AI lies in enabling everyone to become a director or screenwriter, transforming imagined scenes into visuals accessible to all. Genie 4 truly empowers us with directorial control—we no longer need to wait for video generation before making adjustments but can direct characters, props, and even the entire environment in real-time.

‘Real-time interactivity’ is set to become the next battleground in the world model industry. Models like GWM Worlds 2 by startup Runway, JD’s JoyAI-EchoWM, and Alibaba’s Happy Oyster also offer similar capabilities.

It is worth noting that while Genie demonstrates video generation capabilities, using it solely for video would be underutilizing its potential.

‘Real-Time Interactive World Models’ for Robot Training

After Genie 4’s release, Leitech noticed that many reports emphasized its video generation capabilities and linked it to gaming, with some media even proclaiming ‘the death of game engines.’

Indeed, its ability to generate virtual worlds and enable real-time interaction resembles the virtual world games depicted in online novels. Many believe combining this model with VR could immerse us in virtual worlds.

However, in Leitech’s view, Genie 4 or other real-time interactive world models cannot support virtual world games in the near term. Building a virtual world is far more costly than generating a video, and local PC performance is insufficient. Moreover, a game with save functionality, online play, precise collision detection, and reproducible rules demands ‘every frame be calculable’—a task generative models struggle with.

(Image source: Generated by Doubao AI)

While Genie 4 has improved over its predecessor, it can only maintain coherence for up to 2 minutes. For longer tasks, ensuring object consistency becomes challenging.

Leitech believes the first beneficiaries of Genie 4 or similar models will be the film and television industry, where its ‘constrained generation process’ offers significant advantages over traditional video models. Next are industries like news, education, gaming, and engineering, where Genie 4 can provide creative prototyping capabilities, allowing users to quickly transform ideas into explorable prototypes. Finally, the key application lies in embodied AI training.

Take autonomous vehicles as an example: Companies like Huawei and XPENG use world models to generate scenarios for targeted training of intelligent driving systems, enhancing their performance in long-tail and complex scenarios.

(Image source: Leitech)

Real-time interactive world models enable users to introduce ‘unexpected events’ during intelligent driving system training, such as sudden fog or heavy rain, further refining the system’s ability to handle emergencies.

The same applies to robotics: Real-time interactive world models can dynamically generate diverse open scenarios with movable, collidable, and audibly responsive objects based on natural language instructions. Combined with native audio synchronization, they can train multimodal embodied AI (vision + hearing)—for instance, reacting to falling sounds or assessing status based on ambient noise—dimensions traditional simulations struggle to achieve cost-effectively.

Of course, the world simulated by Genie 4 only resembles the real world and still differs from reality. Companies cannot use it as a ‘physical simulation’ in the short term. While training embodied AI with real-time interactive world models, real-world data remains crucial.

Genie 4 Is Just the First Step—The Potential of World Models Awaits

Google’s release of Genie 4 over a year after Genie 3 breaks the limitation of traditional AI videos being generated in one go and difficult to modify mid-process, making real-time interactivity a new competitive direction in video generation. In scenarios like gaming and embodied AI training, it has sparked boundless imagination.

However, Genie 4 faces three challenges: (1) the timing and pricing of access for regular users, which will determine whether it remains a lab toy or becomes a widely used tool; (2) whether long-term consistency can improve beyond 2 minutes; and (3) whether the ‘generate-train-transfer’ loop can succeed in embodied AI, a key test of Physical AI capabilities.

Two years ago, AI struggled to accurately depict hands in static images. Today, it can generate a listening, speaking world in real-time. While Genie 4 currently maintains coherence and object consistency for only 2 minutes, its future is not bound by time.

Genie 4 is not alone. Just as Kling and Jiemeng accompanied Sora, models like GWM Worlds 2, JD’s JoyAI-EchoWM, and Alibaba’s Happy Oyster will grow and evolve alongside Genie. This industry holds infinite potential—after all, the second half of the world model race has only just begun.

Source: Leitech

Images in this article are from 123RF Royalty-Free Image Library. Source: Leitech