"World models" have become a buzzword in artificial intelligence, spanning model-based reinforcement learning, video synthesis, embodied robotics, and the broader vision of "Physical AI." Researchers across disciplines are developing systems labeled as "world models," and even everyday consumers are encountering the term in various contexts.

However, a persistent challenge remains: within academia, there is no unified agreement on the precise definition or internal architecture of "world models." This article leverages the latest cutting-edge research to address three fundamental questions: What exactly is a world model? What variations exist? What capabilities do they offer?

Origins: A 1943 Insight

The concept of world models is not new. As early as 1943, psychologist Kenneth Craik proposed that organisms survive and reproduce by constructing "working models" of the physical world in their brains.

The core idea is that organisms do not rely solely on slow, trial-and-error physical interactions. Instead, the brain simulates hypothetical scenarios internally, predicts the outcomes of potential actions, and pre-computes optimal strategies. This forms the foundation of today's "world models": running simulations of the world in the mind before taking action.

Today, this concept is widely applied in fields like autonomous driving, humanoid robotics, and computer vision.

Classification: Fei-Fei Li's "Tripartite Functional Framework"

Amidst the diverse definitions, Fei-Fei Li, often referred to as the "Godmother of AI," recently proposed a functional classification system for world models. In her latest article, "Functional Classification of World Models," she categorizes them into three types based on their output capabilities:

• Renderer: Generates videos, effectively "painting" the internal world model.
• Simulator: Predicts state transitions, answering the question, "What will happen next?"
• Planner: Directly produces actions, guiding agent behavior.

This classification standardizes the language around what world models can output. However, it has limitations: it specifies only outputs and use cases, not the internal structure, functioning, or architecture of world models. This leads to the next debate.

Architectural Debate: Pixel Reconstruction or Latent Space Prediction?

The internal construction methods of world models remain a topic of intense debate, with two main schools of thought:

• Fei-Fei Li's World Labs: Argues that world models are unified internal models capable of decoding into RGB pixels, state vectors, or candidate action proposals through different query interfaces—one model, multiple outputs.
• The JEPA School (led by LeCun): Contends that generative pixel-level reconstruction is not the ultimate goal. Predicting in latent space suffices, as pixel-level reconstruction wastes valuable representational capacity on irrelevant photometric details. (See "World Models V-JEPA Enters Auxiliary Driving Applications, Poised to Revolutionize Physical AI")

Both schools share a common belief, analogous to large language models (LLMs): The success of LLMs lies in compressing human linguistic knowledge. Similarly, the core goal of world models is to compress the physical laws of the real world (the joint distribution of sensory observations and agent actions). If world models succeed, it means the knowledge of the physical world has been compressed into the model.

Thus, the debate centers on a single question: How should the world be represented, and to what degree should it be compressed? Precise pixel-level reconstruction is not the ultimate goal; the key is retaining sufficient useful information for downstream applications. The challenge lies in balancing high-fidelity physical details with abstract semantics through lossy compression.

Data Determines the Upper Bound: Internet Videos as the Greatest Treasure

The LLM era has proven that data sets the upper bound of AI capabilities. This principle holds equally in the physical world: with fixed architecture and compute, the diversity of physical experiences represented in training data determines the upper bound of generalization. Architecture and compute only affect how efficiently this bound is approached, not the bound itself.

The problem is that data collected by robots and autonomous driving hardware is far from sufficient in scale or diversity. Currently, the only data source that meets demand is the open internet: hundreds of billions of images, videos, and texts.

Internet videos, in particular, are crucial. The raw pixel streams of everyday videos implicitly encode foundational priors governing physical reality:

• Object permanence (objects continue to exist even when out of view).
• Distinctions between rigid bodies and deformable materials.
• Kinematic constraints on motion and interaction.
• Temporal dynamics of lighting and shadows.
• Causal logic of occlusion and de-occlusion.
• Structured priors on human-like behavior and interaction.

However, this knowledge is "hidden in unlabeled pixels" and not directly accessible. The mainstream approach uses a two-stage pipeline:

1. Automatic filtering and annotation extract implicit physical knowledge from massive online videos to establish the upper bound of generalization.
2. Refinement and screening remove irrelevant visual content, synthesize standardized action representations, and retain only motion and interaction signals physically meaningful to robots. This allows the model to converge efficiently on embodied tasks without losing the broad physical priors learned from billions of videos. This logic aligns with LLM training.

Formal Definition: One Model, Three Characteristics

Based on the above discussion, a working definition of world models can be proposed: World models are compressed representations of state transition processes in the physical world, constructed under constraints of limited computational resources.

Under computational constraints, general-purpose physical world models naturally exhibit three characteristics:

• Multimodal: Not limited to text or vision, they must model all perceptual modalities and form unified latent representations.
• Multidimensional and Asynchronous: In reality, sensors sample at different frequencies, so the model must handle multidimensional, asynchronous (multi-frequency) sequential data.
• Locality: An agent's perception is resource-constrained, observing only local regions while the external world continuously imposes interventions on the locality. Thus, modeling is often formalized as a Partially Observable Markov Decision Process (POMDP).

This definition also brings a paradigm shift. Most existing models answer the question, "Given current observations, what will happen next?" A true physical world model, however, must answer, "What is happening, why is it happening, and what will happen?"—requiring cognitive reasoning: identifying discrete operational states of the system, delineating boundaries between normal and abnormal conditions, and tracing the causal trajectory of system evolution.

Why does this matter? Because the physical world differs fundamentally from the digital world: digital data, once generated, is permanently readable. Language models trained years ago can still answer quantum mechanics questions today. In contrast, physical systems are inherently non-stationary and constantly changing. The most critical events (e.g., safety failures, rare fault modes) may never appear in static training corpora.

This is why static world models struggle to cope with the physical world—we need self-consistent dynamic systems that can adjust autonomously and in real-time based on general-purpose representations for specific applications, without collapsing when new modalities are introduced.

Two Perspectives: Tools for Understanding vs. Tools for Prediction

Scholars view world models through two complementary lenses:

• As Understanding Tools: They compress sensory data into stable internal representations, revealing entities, relationships, and mechanisms needed to explain current situations. Prediction here primarily serves as a training signal, forcing representations to learn correct latent structures. Early deep world models (e.g., Ha & Schmidhuber 2018) followed this approach.
• As Prediction Tools: Their value lies in foresight—predicting how the world will evolve, generating candidate future scenarios, and supporting planning and decision-making. LeCun's Autonomous Intelligence framework places predictive world models at the core of reasoning and action, while large-scale video generators like Sora represent this view at the level of "observable futures."

Building on this, the World Action Model (WAM) bridges prediction and action: it can reason using explicit future video (visual planning), compact latent trajectories, or structured spatial representations like optical flow, 3D point flows, or RGB-D trajectories. On Fei-Fei Li's functional axis for world models, WAM is primarily a planner but often doubles as a simulator—since future state prediction is integral to action generation, not merely an auxiliary tool.

If we classify current mainstream world models, the following diagram emerges:

Beyond Robotics: Can World Models Advance Scientific Research?

The preceding discussion focuses on everyday physical environments—robots, vehicles, manipulable objects. Can this framework extend to scientific research? Absolutely, because the underlying paradigm remains unchanged: three elements persist—system states, observable measurements, and implementable interventions. Applied to disciplines:

• Chemistry: States are molecular species, structures, and reaction progressions; observations are spectra, chromatograms, and yields; interventions are controlled experimental manipulations.
• Biology: States span molecular networks to multi-scale organismal processes; interventions include genetic perturbations and drug administration.
• Astronomy: States are indirectly observed through instruments; "planning" manifests as choices of targets, instruments, and observational schedules.

Indeed, existing scientific models already partially embody this framework: GraphCast in meteorology learns autoregressive transition models of global atmospheric states for medium-range forecasting (e.g., predicting typhoons)—a structured scientific simulator. However, it is not yet an agentic world model, as it does not select interventions or form a "prediction-action" loop. Similarly, MDGen at the molecular scale learns generative models of molecular dynamics trajectories but does not constitute a complete experimental planning system.

A full-fledged agentic scientific world model must go further: it should not merely generate plausible measurements but capture mechanisms, adhere to physical constraints and invariants, quantify its own uncertainty, support counterfactual interventions, and integrate these capabilities into an experimental loop—posing hypotheses, predicting outcomes, executing experiments, and updating internal states based on new evidence.

Conclusion

To summarize what world models are, we can condense the full text into three sentences:

• What is a world model?—It is a compressed representation of state transitions in the physical world, constrained by limited computational resources; it is not merely a pixel predictor but an internal simulator encompassing latent states, observation formation, and action-conditioned dynamics.
• How does it grow?—Data diversity determines the upper bound of generalization. Internet videos are currently the only qualified source of physical priors, trained following LLM methodologies (to be shared in subsequent articles).
• Where is it headed?—From answering "what will happen next" to answering "what is happening, why, and what will happen"; from robotics and autonomous driving to closed-loop scientific discovery in chemistry, biology, and astronomy.

Finally, follow Vehicle for upcoming articles delving deeper into the key technologies and architectures of world models.

References and Images

• Understanding World or Predicting Future? A Comprehensive Survey of World Models JINGTAO DING∗ , YUNKE ZHANG∗ , YU SHANG∗ , JIE FENG∗ , YUHENG ZHANG† , ZEFANG ZONG† , YUAN YUAN† , HONGYUAN SU† , NIAN LI† , JINGHUA PIAO† , YUCHENG DENG, NICHOLAS SUKIENNIK, CHEN GAO, FENGLI XU, YONG LI, Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, China.
• A Definition and Roadmap for World Models Physical Intelligence Team, Shanghai AI Laboratory

*Unauthorized reproduction or excerpting is strictly prohibited.