# RoboGate Observatory — Fair Harness Test Protocol **Version:** 2.0 draft **Status:** implementation complete locally; target-GPU qualification pending **Effective date:** none — automatic evaluation and publication remain paused **Supersedes:** Test Protocol v1.0 for all new learned-policy results ## 1. Scope This protocol defines a 68-scenario Franka Panda assisted Pick&Place evaluation in NVIDIA Isaac Sim. It measures one declared model, checkpoint, adapter, imported policy runtime, harness, image, hardware configuration, scenario manifest, seed set, and step budget. A result is evidence only for that declared contract. It is not a general safety certificate or production-readiness guarantee. The task uses the same kinematic grasp attachment for every control and learned policy. It scores reaching, attachment, lifting, transport, release, and stable placement. It does **not** measure payload capacity, grip force, contact safety, or held-object friction dynamics; mass and surface parameters must not be used to make those claims from this benchmark. Learned-policy rows measure **unadapted cross-environment, zero-shot task transfer**. The adapter preserves metric world axes but does not translate the Isaac absolute state origin onto a training-environment origin. The canonical instruction, `pick up the red block and place it on the green target`, is held constant across all 68 scenarios; it is outside the audited GR00T LIBERO training-task set. Both the environment and task distribution shifts are part of the declared scope and must not be generalized into a simulator-only or architecture-wide capability claim. ## 2. Immutable run contract Every run must record and hash: - harness ID, source revision, harness SHA-256 and container digest; - simulator and version, GPU hardware and driver; - scenario manifest SHA-256, ordered scenario IDs and seeds; - model ID, checkpoint path/revision and content SHA-256; - adapter ID, version, implementation SHA-256 and configuration; - imported policy backend, module, class, distribution/version, complete Python package source SHA-256, policy-module SHA-256, and clean Git revision (or installed-package identity when the source is not a tracked checkout); - camera set, action semantics, control settings and fixed step budget; and - run ID, timestamps, host identity and qualification ID. A contract mismatch is an `ERROR`; it cannot become an ordinary episode failure. ## 3. Scenarios and physical application The manifest contains exactly 68 unique scenarios across nominal, edge, adversarial and domain-randomization domains. Every label resolves deterministically to an applied scene specification that includes relevant object, mass, pose, material, lighting, camera, surface, obstacle, occluder, distractor and bounded-disturbance parameters. The server returns full application evidence on every observation. The adapter and result validator independently reconstruct and verify that evidence. Readback schema `1.1.0` canonicalizes numeric values to IEEE-754 binary32, the narrowest precision of the simulator attributes in this contract. Structure, types, canonical values and the readback self-hash remain exact. Derived direction components below `1e-12` are canonical zero to remove double-precision transform noise; any other numerical difference invalidates the run. Authored light orientation uses a double-precision USD quaternion. ## 4. Observation and action interface - Protocol: `2.0.0`; harness ID: `robogate-fair-pnp-v2`. - Observation: independent primary and wrist RGB sensors, 7 joint positions, two gripper positions, end-effector position and orientation, scenario identity, applied-scene evidence, step, done, success and diagnostic info. - Identical primary/wrist frames, blank images, non-finite state, wrong scenario, or wrong protocol are rejected. - The wrist sensor is rigidly parented to Franka `panda_hand` with the robosuite 1.4 Panda `eye_in_hand` local pose and 75° field of view. Its mount contract is immutable and the camera rotates with the hand. - The scored control interface is 20 Hz: three 1/60 s physics steps per policy action. A slower hidden settle period is not permitted. - Action: seven finite normalized LIBERO OSC_POSE values in `[-1, 1]`: world-frame `delta_xyz` scaled to 0.05 m, world-frame axis-angle scaled to 0.5 rad and a gripper command where negative opens and non-negative closes. - This aligns the robosuite 1.4 command mapping, not its controller dynamics. Isaac Lula IK plus joint-position control executes the requested poses, and that controller shift remains part of the cross-environment measurement. - The GR00T LIBERO adapter uses end-effector XYZ, WXYZ-quaternion converted to the official axis-angle representation, and signed two-finger qpos for its 8D state. Both independent images are rotated 180 degrees as in the official LIBERO environment processor. Checkpoint feature, processor and action-decode files are validated before model loading. - The natural-language instruction is immutable and identical across scenarios. Scenario category or variant labels never alter the model prompt. - A client must register its model, checkpoint, adapter and imported-policy source identity before reset. Every observation must echo that exact bound registration; a missing or changed identity invalidates the run. - Inference exceptions, protocol failures, timeouts and zero-action fallback are errors. Live inference calls and action hashes are attested. - A finite in-range action whose requested EE pose is unreachable is a declared `control_rejected` episode outcome, not an infrastructure error or a silent held action. A missing/broken IK solver remains a run-level error. ## 5. Success state machine Object-to-target distance alone is never sufficient. Success requires this ordered physical chain: 1. gripper reaches the object; 2. object becomes grasped; 3. grasped object is lifted above its initial height; 4. grasped object is transported into the target region; 5. object is released by an open-gripper transition; and 6. the released object remains in the goal for the required stable steps. Target entry must be attributed to release. Disturbance motion cannot create policy credit. ## 6. Harness qualification One immutable harness contract qualifies only when: - a zero-action control passes 0 of 68 scenarios; - a perfect positive control passes all 68 scenarios; - at least three full positive-control runs have identical semantic outcomes; - every control artifact passes the same strict result validator; and - all control runs share identical manifest, image, harness, budget and camera fingerprints. The passed qualification document is inseparable from an immutable bundle containing the zero-action result and all three oracle results. Promotion, CI, and downstream dataset sync recompute the qualification ID, checks, semantic signatures, fingerprints, and result hashes from that raw bundle. Qualification is tied to the exact harness and image digest. Any relevant change requires requalification. ## 7. Result completeness A candidate artifact must contain exactly 68 unique, ordered episode records matching the canonical manifest. Each episode carries scenario parameters, applied-scene evidence, transitions, outcome, error state, timing, diagnostics and seed. Summary counts must recompute from the episodes. Episode and document hashes must verify. The imported policy package source and selected module are hashed after model loading and again after all episodes. A tracked checkout must contain exactly the tracked Python source set and be clean. Any source mutation or identity change invalidates the run before a scored artifact can be written. Only a result referencing a current passed qualification record is eligible for promotion. ## 8. Publication The publisher accepts only a promoted evidence bundle containing the raw result, raw qualification controls, recomputed qualification record, manifest, provenance and hashes. Promotion and public replacement are atomic. Missing, malformed, incomplete, unqualified or tampered artifacts remain `ERROR` or `QUARANTINED`.