Over the past two years, the term 'world model' has emerged as one of the most talked-about yet perplexing concepts in the AI landscape.

At the recent GTC conference, Jensen Huang positioned 'physical AI' and 'world models' as the cornerstone of next-generation AI. Autonomous driving firms consider world models indispensable for advancing intelligent driving systems. Robot manufacturers assert that their world models endow robotic arms with versatile manipulation capabilities. Even video generation teams have rebranded their latest offerings as 'world models,' emphasizing that video creation is synonymous with modeling reality.

The 'world model' is rapidly becoming AI's new 'catch-all term,' akin to the metaverse frenzy of a few years back or the large model trend of last year. Any technology bearing this label instantly garners attention from investors and the public alike.

Behind this trend lies a collective industry anxiety over the diminishing returns of large language models. With internet text data nearing exhaustion and the novelty of generative content fading fast, AI urgently seeks its next trillion-dollar application. This necessitates a shift from the digital realm to the physical world, from information processing to tangible manipulation.

Precisely because of its vast potential, the world model faces significant hurdles before becoming truly practical. It's not merely a matter of a single technological breakthrough but a complex puzzle involving conceptual clarity, data acquisition, and architectural design. Each layer of challenges reveals a cognitive gap that AI must bridge to transition into the physical world.

The first major hurdle for world models lies in their very definition—the concept itself.

Numerous video generation models, 3D reconstruction tools, and multimodal large models have adopted this label, yet the industry lacks consensus on what constitutes a world model, its technical pathways, or evaluation criteria. The same term is used to describe vastly different technologies across companies.

This confusion is no accident. Players from different sectors leverage the narrative of world models to stake their claim in the next generation of AI technology.

For content generation firms, framing video creation as a world model transforms the traditional AIGC narrative into a more captivating story of interactive world generation, instantly boosting their valuation prospects.

For robotics companies, the world model specifically refers to the ability of robotic arms to model the physical properties, spatial positions, and interaction feedback of objects they manipulate. The goal is to establish a technological edge in a market saturated with similar hardware.

For autonomous driving firms, the world model involves real-time prediction of traffic participants' movement trajectories and environmental changes. It's a core tool for upgrading from passive perception to active anticipation, a key element in advanced intelligent driving narratives.

For computational power providers like NVIDIA, the world model is a foundational model built on simulation platforms like Omniverse, integrating the full chain of 'perception-simulation-planning' and offering universal modeling capabilities to the entire industry.

Different business objectives have thus imbued the same term with vastly different technological connotations.

To clarify this concept, Fei-Fei Li attempted to establish an analytical framework for this chaotic landscape in a comprehensive article published in June this year. She categorized existing world models into three types:

Renderers focus on 'looking realistic,' generating visually appealing pixels and videos without guaranteeing physical or geometric accuracy. Typical examples include Google Genie and OpenAI Sora, with core metrics being visual realism and spatiotemporal coherence.

Simulators pursue structural precision, outputting geometric data, material parameters, and collision meshes rather than visuals. Examples include NVIDIA Omniverse's physical simulation module and Unity PhysX AI, serving as core foundations for digital twins and industrial simulations.

Planners bridge perception and action, enabling agents to anticipate world changes before acting. Examples include trajectory prediction networks in autonomous driving and robot motion planning models, directly serving agent decision-making.

However, this classification itself underscores the issue: when a technological concept requires lengthy explanations to define its boundaries, it means it is far from reaching a stage of technical convergence.

Before 2012, deep learning also went through a phase of multiple competing routes, but ultimately, data and computational power determined the winners. World models, however, still lack unified benchmarks: video generation models use FVD (Fréchet Video Distance) and CLIP scores to measure effectiveness; robotics models use grasping success rates and task completion rates to verify capabilities; autonomous driving models use trajectory prediction errors and disengagement rates to assess value. Without a unified standard, there is no coordinate system for technological iteration. This conceptual chaos is likely to persist for a long time.

Training large language models relies on nearly inexhaustible data: web pages, books, papers, and posts from the internet can be crawled and used with minimal annotation costs. However, training a model capable of understanding the physical world requires an entirely different set of data—multimodal interaction data with precise geometric annotations, physical parameters, and action labels.

The root of the gap lies in the completely different dimensions of information. Text consists of discrete, standardized, unimodal symbols with relatively fixed meanings, making annotation simple. In contrast, the physical world is continuous, high-dimensional, and causally coupled. Consider the simplest action of 'picking up a paper cup.' It involves dozens of physical quantities, including visual texture, spatial depth, finger force, cup deformation, friction coefficient, and motion acceleration. All data must be temporally aligned at the microsecond level; otherwise, it loses training value.

More critically, even if data is collected at great expense, it may not be the right data.

The cost of collecting real-world physical data is staggering. For autonomous driving, each test vehicle's LiDAR, cameras, IMUs, and other hardware cost over a million yuan. Collecting one hour of real-world road test data, combined with annotation and vehicle maintenance costs, can reach thousands of yuan. Covering long-tail scenarios like rain, night, construction zones, and irregular obstacles requires millions of kilometers of road test data, resulting in astronomical overall costs. The robotics field faces even higher costs: Figure AI disclosed that collecting one hour of real-world operational data for humanoid robots costs thousands of times more than text data for large language models, not to mention the additional risks of hardware damage and safety incidents.

Currently, world models are primarily implemented in specific fields like autonomous driving and electronic gaming, with data scale and diversity insufficient to support a general-purpose model. The real physical world offers infinite scenarios: objects under different lighting, friction under varying wear levels, collision effects at different angles... These long-tail scenarios are crucial for testing a model's generalization capabilities but can never be fully covered through collection alone.

Synthetic data was once seen as a solution. Virtual data generated in bulk using physical simulation engines or game engines costs far less than real-world collection.

Currently, the industry has formed three mainstream approaches:

The first involves using classic physics engines like MuJoCo, Bullet, and PhysX to generate standardized kinematic data, such as DeepMind's DM Control Suite and OpenAI Gym.

The second is domain randomization, which improves model generalization by randomly varying lighting, textures, and physical parameters in simulated environments. A classic example is OpenAI's Dactyl robotic hand, which trained in simulations with randomized friction coefficients and lighting conditions, ultimately completing a Rubik's Cube rotation task in the real world.

The third approach involves using generative AI to enhance realism, with diffusion models generating lifelike textures to narrow the visual gap between simulated and real-world data. For example, NVIDIA's Drive Sim uses this technology to enrich simulation scene details.

However, the pitfalls of this approach run deeper than many anticipate.

Many believe the gap between simulation and reality lies in visual realism, but this is not the case. The true gap is the shift in physical distributions: friction coefficients, elastic moduli, and air resistance in simulation engines are idealized values set manually. In the real world, physical parameters are continuously varying and interdependent. For instance, the friction of a rubber piece changes with temperature and wear level—nuances that simulations cannot fully replicate.

Industry tests have shown that robotic models achieving 98% grasping success rates in simulated environments often drop below 60% in real-world transfers. Even with domain randomization optimizations, breaking the 85% success rate barrier remains challenging, with the remaining gap requiring real-world data fine-tuning to fill.

Current popular approaches in the industry, such as real-world data loops and hybrid schemes of 'synthetic pre-training + real-world fine-tuning,' only alleviate contradictions without fundamentally solving the problem.

Growth in the digital world follows a pattern where marginal costs approach zero, while costs in the physical world require substantial real-world investment at every step. This fundamental conflict is an insurmountable obstacle for world models.

Even with a unified definition and sufficient data, a deeper question remains: What architectural approach should we use to build a world model?

This question essentially asks at what level AI should 'represent the world'—through pixels, geometric structures, or abstract states? Different answers point to entirely different technical paths, reflecting distinct underlying philosophies.

The first path, represented by Google's Genie3, is the pixel interaction route, based on the logic that 'vision defines existence.'

These models typically use spatiotemporal diffusion architectures, adding temporal attention mechanisms to traditional image diffusion to learn motion continuity between video frames. Genie 3, for example, supports multi-conditional inputs like text, images, and action commands, generating 1080P interactive videos. Users can control characters via keyboard or mouse, with the model generating logically consistent subsequent frames in real time, creating a highly immersive experience.

This approach offers rapid monetization opportunities, directly applicable to gaming, content generation, and digital human scenarios. It also has low training data thresholds, as internet videos serve as abundant training material, enabling fast model iteration.

However, its drawbacks are critical. Pixel-level fidelity does not equate to physical accuracy. While it can generate a cup shattering, it cannot explain why fragments fly in a particular direction or predict different rebound effects on various surfaces. Industry tests show that videos generated by Sora frequently contain physical errors like object interpenetration, momentum violations, and lighting inconsistencies. Using such models to guide robots would be like asking someone who has only watched movies to operate a machine tool—it may look convincing but is prone to catastrophic failures.

The second path, exemplified by Fei-Fei Li's World Labs' Marble model, is the spatial structure route, based on the assumption that 'structure precedes physics.'

This approach focuses on reconstructing precise 3D spatial structures from visual inputs, using geometric representations instead of pixels. The Marble model can generate semantically labeled, exportable 3D mesh environments from multi-view image inputs, supporting agent navigation and interaction planning. Its 3D assets can be directly imported into game engines.

In autonomous driving, this route has achieved scalable mass production. Occupancy networks, now standard in advanced intelligent driving systems, construct real-time 3D voxel spaces of the surrounding environment from multi-camera visual inputs, identifying obstacle-occupied regions and drivable spaces without relying on high-definition maps. Tesla, XPENG, and Li Auto's latest driving systems all incorporate spatial world models based on occupancy networks, effectively detecting irregular obstacles and construction barriers that traditional perception methods often miss.

However, its limitations are evident. 3D structures represent only the static skeleton of the physical world, while the core value of world models lies in predicting dynamic changes. While spatial routes can tell you a slope's angle, they cannot directly calculate a sphere's rolling acceleration; they can reconstruct an object's shape but struggle to simulate soft-body or fluid deformations. The engineering journey from static structures to dynamic physics remains long.

The third path, represented by Yann LeCun's JEPA architecture, is the cognitive representation route, based on the logic that 'abstraction defines cognition.'

It does not generate pixels but predicts abstract world states, theoretically closest to the human brain's 'mental model.' When walking, we do not render visual scenes mentally but anticipate ground conditions; when throwing, we do not calculate pixel motions but judge force and landing points.

Technically speaking, JEPA employs an encoder to map input images into a high-dimensional latent space. Subsequently, it utilizes a predictor grounded in historical states to forecast future representations within this latent space—all achieved without the need for pixel generation. This approach markedly diminishes computational demands and directs the model's focus toward semantic-level causal patterns.

DreamerV3, which adopts a comparable methodology, has emerged as a mainstream solution for robot reinforcement learning. It constructs a world model within the latent space to anticipate environmental feedback under various actions, thereby facilitating decision-making without real-world interactions and significantly enhancing sample efficiency.

This pathway is perhaps closest to the essence of general intelligence, yet it also seems the most 'elusive.' Latent space representations function as black boxes; it remains ambiguous what the model genuinely comprehends, rendering debugging challenging when errors arise. More critically, there exists no mature, general-purpose solution for translating abstract states into precise motor control commands or for integrating them with low-level motion planning systems. Most implementations remain confined to task-specific end-to-end training with limited generalization capabilities.

Each of the three routes boasts theoretical underpinnings but also harbors inevitable shortcomings. A growing consensus within the industry suggests that the ultimate general-purpose world model will likely integrate elements from all three: leveraging 3D structures as a framework, physical engines for constraints, and abstract representations for decision-making. NVIDIA's World Foundation Models and Tesla's FSD end-to-end systems are already venturing into multi-route integration. However, aligning and efficiently coordinating pixel, geometric, and latent space representations—three entirely disparate systems—poses a challenge of global proportions.

How far removed is the world model from the real world? There exists no straightforward answer.

At the very least, for the next three to five years, world models will continue to undergo iterative evolution. The current landscape resembles that of deep learning around 2012, characterized by severe data silos, undecided technical routes, and benchmarks still in the process of being established—the 'ChatGPT moment' has yet to materialize.

Nevertheless, world models may confront even greater challenges than deep learning did. Deep learning primarily deals with pattern recognition, extracting statistical regularities from data. Its success stemmed from a convergence of computational power, data, and architecture—essentially a triumph of statistical fitting. World models, however, must grapple with causal reasoning, comprehending why objects move as they do and why events unfold in certain ways.

The disparity between these two capabilities transcends mere years of technological iteration; it represents a fundamental shift in cognitive paradigms. Statistical fitting can achieve breakthroughs by amassing computing power and data, but causal reasoning necessitates a bottom-up abstraction of the underlying laws governing world operations. This endeavor is closer to the core of human intelligence and, consequently, far more arduous to achieve.

Of course, we should not dismiss the value of general world models simply because they remain distant aspirations. During the journey toward generalization, implementations in vertical scenarios will likely emerge first. In the realm of autonomous driving, world models will initially achieve accurate predictions of traffic participants' behaviors in specific scenarios, such as highways and urban areas, thereby enhancing system safety and comfort. In industrial robotics, world models tailored to specific workstations will enable robots to adapt to new tasks more swiftly, reducing debugging costs for production lines. In gaming and digital twin domains, interactive world generation will achieve commercialization first, providing financial feedback for subsequent technological iterations. The data in these vertical scenarios is manageable, and the demands are clear, making them the 'training grounds' for technological iterations of world models.

From a long-term perspective, we should not become overly fixated on the label of 'world models' or on a specific technological path. What truly matters is whether we can empower AI to transition from 'seeing the world' to 'understanding the world,' from pixel-level simulation to causal-level reasoning.

Once this milestone is surpassed, AI will transcend its role as a mere content tool in the digital world and evolve into a universal decision-maker in the physical realm. At that juncture, it will not merely represent the innovation of a single product but will signify a bottom-up restructuring of entire industrial systems, transportation systems, and modes of production. In this light, all the conceptual confusion, data dilemmas, and architectural disagreements we face today are necessary preludes to the dawn, and all the fog will eventually be gradually dispersed by the relentless march of technology.