What is Spatial AI?

Spatial AI refers to systems that understand and reason about three-dimensional space and the physical world. Unlike models that work with text or flat images, spatial AI processes depth, geometry and physical relationships.

From pixels to physical understanding

Traditional computer vision identifies objects in photos: a chair, a table, a door. Spatial AI goes further. It knows how far away the chair is, whether the table is stable, and whether a robot could fit through the door. It reconstructs scenes in 3D, predicts how objects move and interact, and reasons about forces, contact and support.

These capabilities matter when software controls something physical. A warehouse robot picking items from a shelf, a drone navigating a construction site, or an autonomous vehicle merging into traffic all rely on accurate spatial understanding. Mistakes in spatial reasoning do not just produce bad predictions—they cause collisions, dropped objects and failed tasks.

What spatial AI models do

Spatial AI systems take in data from the physical world—LiDAR scans, depth cameras, stereo images, CAD files or robot sensor streams—and produce structured representations of that world. Common tasks include:

• 3D reconstruction: converting 2D images or point clouds into complete 3D models of objects and scenes.
• Scene understanding: identifying not just what objects are present, but how they relate spatially (on top of, inside, supported by).
• Structural parsing: breaking down a complex assembly into its parts and understanding how those parts connect.
• Physical reasoning: predicting stability, contact forces and whether a structure will hold together under load.
• Path planning: finding routes through space that avoid obstacles and respect physical constraints.

World models and foundation models for robotics

A world model is a spatial AI system that predicts how the physical world evolves over time. Given sensor data and a proposed action, a world model forecasts what happens next: where objects move, what falls, what remains stable. These models help robots simulate outcomes before acting, reducing trial-and-error in the real world.

Foundation models for robotics apply large-scale machine learning to spatial tasks, aiming to generalise across different scenes, objects and sensor types. Examples of research in this space include systems that reconstruct geometry from images, predict object affordances or plan manipulation tasks. Independent evaluation is essential because a model's ability to generalise from lab data to real-world edge cases determines whether it can be deployed safely.

The sim-to-real gap

Many spatial AI models are trained and tested in simulation. Simulated environments are clean, controlled and easy to label. Real-world data is messy: lighting changes, surfaces are worn or dirty, objects are non-standard, and sensors produce noise.

The sim-to-real gap describes the performance drop when a model trained on synthetic data encounters real scenes. A model might score well on a simulated benchmark but fail on a cluttered factory floor. Bridging this gap requires testing on real-world data—preferably data the model's developers did not select.

Why independent benchmarking matters

Spatial AI is moving from research prototypes into robots, buildings and vehicles. Claims about model performance often come from the teams who built them, measured on benchmarks they chose. Independent testing on verified real-world cases provides a neutral reference, helping users compare models and identify which systems are ready for deployment.

Sky3 is designed to provide that testing layer. Contributors will submit hard real-world cases, reviewers will verify them, and models will be scored in public on geometric accuracy, structural integrity and physical reasoning. The goal is a benchmark that reflects the conditions models will face when deployed, not the conditions their developers preferred.