World Models and How They Are Evaluated

A world model is a system that predicts how the physical world changes over time. Given the current state and a proposed action, it forecasts what happens next—where objects move, what falls, what stays stable.

What world models do

World models learn representations of physical dynamics. They take sensor data—images, point clouds, depth maps—and build an internal model of the scene. When asked "what if I push this object?" or "what if the robot moves left?", the world model simulates the outcome without requiring real-world trial and error.

This predictive capability is valuable for robotics, autonomous systems and industrial automation. A robot using a world model can evaluate multiple strategies in simulation before committing to an action, reducing mistakes and improving task success rates.

Types of world models

World models vary in scope and representation:

• Geometric world models: focus on spatial structure—object positions, shapes, and how they occlude one another. These models predict where things are, not necessarily how they behave.
• Physical world models: incorporate dynamics—forces, friction, gravity, collisions. They predict motion, stability and contact.
• Learned world models: use neural networks trained on data, learning to predict future states from observations without explicit physics equations.
• Hybrid models: combine learned representations with physical constraints or simulators, aiming for both generality and accuracy.

How world models are evaluated

Evaluating a world model means testing whether its predictions match reality. Common evaluation methods include:

Prediction accuracy

Does the model correctly predict future sensor observations? Given the first few frames of a robot's trajectory, can it predict the next frames accurately? Metrics include pixel-level error for images, point cloud divergence for 3D scenes, and trajectory error for predicted paths.

Physical plausibility

Do the predicted states obey basic physical laws? Objects should not float without support, penetrate solid surfaces or accelerate without force. World models are tested on whether they produce physically consistent outcomes or hallucinate impossible configurations.

Generalisation

Can the model handle scenes and objects it has not seen before? Evaluation datasets test models on out-of-distribution cases: unusual object shapes, rare lighting conditions, novel materials or configurations that were absent from training data.

Task performance

Does using the world model improve downstream task success? For manipulation, navigation or planning tasks, the model's utility is measured by whether the robot succeeds more often when using the model's predictions versus baseline methods.

Challenges in evaluation

World model evaluation is difficult because real-world outcomes are complex and context-dependent. The same push on an object produces different results depending on friction, weight distribution and nearby supports. Benchmark datasets often simplify these factors, but deployed systems face the full complexity.

Another challenge is data scarcity. Collecting verified real-world interaction data—complete with before-and-after 3D scans, force measurements and failure modes—is expensive. Most world models are tested on simulated or small-scale lab datasets, which do not fully capture real-world edge cases.

Sky3's approach to evaluating world models

Sky3 will build evaluation datasets from contributed real-world spatial data, focusing on the cases models find difficult: cluttered scenes, non-standard assemblies, ambiguous structures and edge geometries. World models and robotics foundation models will be scored on:

• 3D reconstruction accuracy: how well the model reconstructs spatial structure from partial observations (measured by IoU and other metrics detailed on the metrics page).
• Structural integrity: whether the model preserves load-bearing elements and connections.
• Assembly tree accuracy: how well the model understands part hierarchies and how components fit together.
• Physical logic checks: whether predicted configurations respect stability, support and contact constraints.

These evaluations will be run on verified real-world cases that the model's developers did not select, providing an independent reference for model performance.