Decentralized Trust for Spatial AI
Sky3 is an open, community-governed network for testing spatial and physical-world AI models against real-world data. Contributors will supply hard cases from the physical world, independent reviewers will verify them, and models will be scored in public.
Why spatial AI needs a neutral referee
Models that reason about the physical world are moving from research into robots, factories and buildings. Today, the claims about how well they work mostly come from the people who built them, measured on benchmarks they chose.
Simulation does not capture everything. A model that scores well on synthetic scenes can still misread a cluttered shelf, a non-standard fastener or a structure that only holds together because of one hidden joint. These gaps show up when a system is deployed, and that is the most expensive place to find them.
Sky3 proposes a simple alternative: test models on real-world cases that the model's builders did not pick, and publish the results.
- Independent. Benchmarks are run by the network, not by the teams whose models are being scored.
- Grounded in the real world. Test cases come from physical scans, drawings and recordings, with a focus on rare and difficult cases.
- Verified. Every contributed case is checked by automated tests and by human reviewers before it is used for scoring.
- Reproducible. Metrics, scoring code and evaluation splits are published so anyone can check a result.
- Private by default. Submissions are stripped of metadata, faces and identifying information before they enter the archive.
- Community-governed. The test suite and the rules for changing it are governed by contributors and reviewers through an open process, stewarded by a non-profit foundation.
This is a draft. The protocol, metrics and governance model will change based on feedback from the beta.
What a public scorecard looks like
The leaderboard will rank spatial and physical-world models on verified real-world cases. The table below uses made-up models and made-up numbers to show the format.
All model names and scores are invented for illustration. No real model has been evaluated.
From real-world data to a public score
Contribute real-world edge cases
Contributors will submit spatial data that models find hard: LiDAR scans, CAD files, robot trajectories and images of real geometry. Rare and complex cases are worth more. Accepted submissions will earn points that record each contributor's share of the archive.
- Planned formats: point clouds (.las, .laz, .ply), CAD (.step, .stl, .obj), trajectories (.csv, .json), images (.jpg, .png).
- Files will be sanitized on intake: metadata stripped, faces and personal information removed.
Verify every case before it counts
Each submission will go through automated checks for format, duplication and quality, then review by independent human verifiers. A case will enter the benchmark only when reviewers agree on its labels. Reviewers will earn points for accurate reviews.
- Automated checks: file integrity, duplicate detection, label consistency.
- Human review: multiple independent reviewers per case, agreement required.
Score models in public
Verified cases will form held-out evaluation sets. Models will be scored on geometric accuracy, structural understanding and physical reasoning, and results will be published on the Spatial Leaderboard with the method used to produce them.
- Metrics include IoU, structural integrity, assembly tree accuracy and physical logic (defined in the leaderboard).
- Model developers will be able to submit through a sandbox API (planned).
Points are a record of contribution during the beta. They have no monetary value, and any future use will be decided through the governance process.
Join the beta
We're opening the first data-ingestion beta to a small group of contributors, AI developers and researchers. Leave your details and we'll email you when there's a place.
You're on the list.
We'll email when the beta opens. No other emails before then.
Something went wrong saving your details. Please try again.