Baseline
Instruction: Return exactly WRONG for each input.
Recorded result: fail · 0/1
Review a recorded skill comparison
Two model-backed runs used one synthetic case in a disposable local test. The result is inconclusive.
The case
This first-party test used a synthetic acknowledgement task. The model received the same case for both skill revisions.
- Input
- acknowledgement test
- Expected output
- ACK
The model judge was instructed to award one point only when the response is exactly ACK.
Compare the runs
The test changed the skill instruction. It used the same model and reserved case for both runs.
Candidate
Instruction: Return exactly ACK for each input.
Recorded result: pass · 1/1
The raw model responses and judge output were not retained. The receipt records the outcomes, but it does not permit independent inspection of the responses.
The decision
Inconclusive
The recorded 95% interval for the change is [-1, 1]. It includes zero. One case does not establish general improvement.
A scripted test reviewer accepted the candidate with uncertainty acknowledged. This was not a human review or a customer release decision.
Inspect the record
- Model
- google-vertex:gemini-2.5-flash
- Recorded at
- Case count
- 1
- Test harness revision
- 3fd0fbc2d328175cc861e2adb0d6727c91e99e2f
- Receipt SHA-256
- 13fb7f7c110594b7e993084a9c4cb24bed60a4e4ecaf1498ac2d03f852f66564
Open the JSON receiptRead the agent guideTry a local code check
The test ran in a disposable local environment. The receipt is a model-backed skill execution, not an Assurance release-review record or a tool-using agent run. It is not a customer or production result.