Evaluation Metrics

Sky3 scores spatial AI models on geometric accuracy, structural understanding and physical reasoning. This page explains each metric, how it is calculated, and which are standard industry metrics versus Sky3's proposed protocol metrics. For context on related public benchmarks, see the benchmarks directory.

2D-to-3D Construction Accuracy

Type: Standard metric category (Sky3 applies it to real-world edge cases)

2D-to-3D construction measures how accurately a model reconstructs three-dimensional structure from two-dimensional inputs such as images, sketches or technical drawings. This is a fundamental task for spatial AI: converting flat representations into volumetric models.

Common input scenarios include:

• Multi-view imagery with known or estimated camera poses
• Single-view reconstruction where the model infers depth and occluded geometry
• Technical drawings or CAD sketches that specify dimensions and relationships
• Hybrid inputs combining images with partial 3D data (sparse point clouds, depth maps)

Sky3 will evaluate 2D-to-3D construction by comparing the predicted 3D output to verified ground truth captured via LiDAR, photogrammetry or high-precision CAD. Accuracy will be measured using geometry metrics (IoU, Chamfer distance, surface normals) and completeness checks (coverage of visible and occluded regions).

Why it matters: 2D-to-3D construction is a core capability for robotics (understanding scenes from camera input), autonomous systems (mapping environments), and CAD automation (converting sketches to models). Models that fail on real-world edge cases—unusual perspectives, occlusions, non-standard objects—cannot be trusted in deployed systems.

Bounding Box Metrics (2D and 3D IoU, mAP)

Type: Standard metrics

Bounding boxes are the simplest spatial abstraction: axis-aligned or oriented boxes that enclose objects. Bounding box metrics measure how well a model detects and localizes objects in 2D images or 3D scenes.

2D Bounding Box IoU

For 2D object detection in images, IoU is computed over rectangular regions:

where P is the predicted bounding box and G is the ground truth box. An IoU threshold (commonly 0.5 or 0.75) determines whether a detection is considered correct.

3D Bounding Box IoU

For 3D object detection in point clouds or reconstructed scenes, IoU is computed over three-dimensional volumes. For axis-aligned boxes:

For oriented bounding boxes (OBB), the boxes may be rotated to fit the object's principal axes. Intersection and union volumes are computed in 3D space.

Mean Average Precision (mAP)

mAP summarizes detection performance across multiple object classes and IoU thresholds. For each class:

1. Rank detections by confidence score.
2. Compute precision and recall at each rank by comparing to ground truth using an IoU threshold.
3. Calculate Average Precision (AP) as the area under the precision-recall curve.
4. Average AP across all classes to produce mAP.

Sky3 will report mAP at IoU thresholds of 0.5 and 0.75 for both 2D and 3D detection tasks. Higher mAP indicates better localization and fewer false positives.

Why it matters: Bounding box detection is the foundation for scene understanding. Robotics, warehouse automation, and autonomous vehicles rely on accurate object localization. Errors in bounding boxes propagate to downstream tasks like grasping, path planning and collision avoidance.

Structural Component Parsing

Type: Standard task category (Sky3's integrity metric is a proposed extension)

Structural component parsing refers to segmenting a scene or object into meaningful parts: walls, beams, fasteners, panels, joints. Beyond detecting that components exist, parsing identifies their role in the structure.

Evaluation metrics for parsing include:

• Part-level segmentation accuracy: IoU or F1 score computed per part category (standard semantic segmentation metrics).
• Instance segmentation: distinguishing individual instances of the same part type (e.g., multiple bolts).
• Hierarchical correctness: whether the model correctly identifies subassembly relationships (see Assembly Tree Accuracy below).

Why it matters: Parsing is essential for tasks like disassembly, repair, inspection and manufacturing planning. A model that reconstructs geometry but cannot identify which parts are load-bearing or how they connect is not useful for real-world engineering applications.

Volumetric 3D IoU

Type: Standard metric

Volumetric IoU measures how well a predicted 3D reconstruction matches the ground truth across the entire volume. It is the ratio of the overlap volume to the total volume covered by either the prediction or the ground truth.

where:

• P is the predicted volume (bounding box, mesh or voxel grid)
• G is the ground truth volume
• V(P ∩ G) is the volume of intersection
• V(P ∪ G) is the volume of union

IoU ranges from 0 (no overlap) to 1 (perfect match). Sky3 will report mean IoU across all verified test cases. For scenes with multiple objects, per-object IoU will be computed and averaged. For voxel grids, intersection and union will be counted at the voxel level.

Why it matters: IoU is a standard measure of spatial accuracy in 3D reconstruction and object detection. Higher IoU means the model's geometry is closer to the real structure.

Structural Integrity

Type: Sky3 proposed metric

Structural integrity measures whether a reconstructed assembly preserves load-bearing components and connections. A model may achieve decent IoU while missing critical fasteners, joints or support elements that hold a structure together. This metric checks for those omissions.

Calculation:

1. Identify load-bearing parts and connections in the ground truth (verified by human reviewers or structural simulation).
2. Check which of those elements are present in the predicted reconstruction.
3. Compute the percentage of critical elements correctly detected.

A structure scores high on this metric only if it retains the components necessary for physical integrity. Missing a decorative panel is less penalised than missing a load-bearing beam or a connecting bolt.

Why it matters: For robotics and industrial applications, understanding which parts are essential is as important as reconstructing overall geometry. A model that ignores fasteners or connectors will produce assemblies that do not hold together.

Assembly Tree Accuracy

Type: Sky3 proposed metric

Many physical structures are hierarchical assemblies: a product is made of subassemblies, which contain parts, which may themselves be composite. Assembly tree accuracy measures how well a model captures this hierarchy.

The ground truth assembly tree is represented as a graph where nodes are parts or subassemblies and edges represent "contains" or "attaches to" relationships. The model's predicted tree is compared to this reference using tree edit distance or node-and-edge matching.

Tree Edit Distance approach:

We then compute accuracy as:

Node-and-Edge F1 approach (alternative):

Compute precision and recall for both nodes (parts correctly identified) and edges (relationships correctly identified), then take the F1 score:

Edge F1 is computed similarly. The final assembly tree accuracy is the average of node F1 and edge F1, scaled to a percentage.

Why it matters: Understanding part hierarchies is critical for tasks like disassembly, repair and manufacturing planning. A flat list of parts is less useful than a structured tree showing how they connect.

Physical Logic Checks

Type: Sky3 proposed metric

Physical logic checks test whether a predicted reconstruction obeys basic physical constraints. The protocol will run a suite of automated tests on each reconstruction:

Stability check

Does the structure have a stable center of mass and base of support? The test will compute the center of mass assuming uniform density and check whether it falls within the support polygon defined by contact points with the ground or supporting surfaces.

Support check

Are objects resting on appropriate supports? Floating objects (not in contact with a surface and not designated as suspended) will fail this check.

Contact check

Do connected parts actually touch or overlap? The test will verify that parts declared as connected have non-zero contact area in the predicted geometry.

Penetration check

Do solid objects improperly penetrate each other? Small overlaps (within tolerance for meshing artifacts) are allowed, but significant interpenetration indicates a geometric error.

The physical logic pass rate is the percentage of these checks that the reconstruction passes:

Why it matters: Geometric accuracy is not enough if the result is physically impossible. A model that predicts structures with floating parts or interpenetrating solids has not learned correct physical reasoning.

Overall Score

The overall score is the unweighted mean of the four normalised metrics:

This gives a single number summarising model performance across geometric, structural and physical dimensions. All four metrics are equally weighted in the current protocol.

Metric limitations and future work

These metrics are part of Sky3's draft protocol. They are designed to capture dimensions of spatial understanding that matter for real-world deployment, but they are not exhaustive. Planned extensions include:

• Material property recognition (identifying which parts are rigid, flexible, brittle).
• Temporal prediction accuracy for world models (how well they predict motion over time).
• Task-specific scoring for manipulation, navigation and assembly planning.
• Robustness metrics measuring performance degradation under sensor noise or occlusion.

The protocol and metrics will evolve based on feedback from the beta and the research community. Changes will be documented and versioned, and historical scores will be preserved for comparison.