Baseline policy
Identify the category as access or billing. Assign normal priority to every request. Return no other category or priority.
Review a recorded coding-agent comparison
A first-party local test compared two policies on four synthetic support requests. The result is self-reported, not a hosted Assurance review.
The task
The coding agent classified synthetic support requests by category and priority. The requests contain no customer data.
Candidate policy
Identify the category as access or billing. Assign high priority when a request says that a user cannot sign in. Assign normal priority to billing requests. Return no other category or priority.
The agent read the inputs and wrote output.json in each completed run.
The public JSON record was assembled from local command-line interface (CLI) and verifier records and preserved output files. The tool activity summary was derived from local event logs. The run metadata is not a signed provider receipt.
Compare the results
Baseline
Cases passed: 2/4
CLI verifier score: 0
Candidate
Cases passed: 4/4
CLI verifier score: 100
The local verifier gives 100 when every case passes and 0 otherwise. These scores are not case-pass percentages. The CLI reported improved. No human release decision was recorded.
Inspect each case
case-1
All staff are locked out and cannot sign in.
- Expected
access·high- Baseline
access·normal· fail- Candidate
access·high· pass
case-2
Please send a copy of last month's invoice.
- Expected
billing·normal- Baseline
billing·normal· pass- Candidate
billing·normal· pass
case-3
I cannot sign in to my account.
- Expected
access·high- Baseline
access·normal· fail- Candidate
access·high· pass
case-4
Please update the billing address for next month.
- Expected
billing·normal- Baseline
billing·normal· pass- Candidate
billing·normal· pass
What this result can and cannot show
- Four synthetic cases were used to design the candidate policy. There was no independent holdout set.
- The CLI uses an all-or-nothing local verifier and does not independently attest provider identity or execution.
- The baseline run warned that some skill descriptions were shortened. The candidate did not report that warning, so all hidden runtime context is not proven identical.
- A first attempt using a different model was rejected before execution. It is not part of this pair.
- No human reviewer recorded a release decision. This does not establish general improvement.
- Raw agent event logs are not published. The tool activity summary is derived from local logs.
Inspect run diagnostics
baseline
- The local model cache could not load.
- An installed plugin could not load.
- The local thread index reported a state database discrepancy and used a fallback.
candidate
- An installed plugin could not load.
- The local thread index reported a state database discrepancy and used a fallback.
Inspect the record
- Agent client
- Codex CLI 0.145.0
- Requested model
- gpt-5.6-sol · not independently attested
- Recorded at
- Baseline run
- run-20261027025539206-4444d30d
- Candidate run
- run-20261027025608665-8e0cb1f3
- Challenge SHA-256
- 129acbdca4c667cf2828c584f12c778e7f4fb26603bbc979f544523d317c90e2
- Cases SHA-256
- 9e5bf4c0465673844a3f1c569d67ffc30ab37d282edca1ce4418d0c35dd7310b
- Expected results SHA-256
- 92bed1342f50e7aca63f463049aa5c5c4c900a7e55271f95fab55dbb7b8a0040
- Verifier SHA-256
- aee889e4e3a7ba64dc34eb213130b18371924380fc194f40dcd12f3cbe3d3b3c
- Baseline output SHA-256
- 2051eaf8cba8f60e4a98a76582ec344df1b2f1b321c3b26d4f7a871d889c6a88
- Candidate output SHA-256
- b72b0951d812407382bcf7550dc2ba47bfb366fd26e17136d99508cf0612da85
- Record SHA-256
- 3c4f9d11189a4b0e941df12d8aeffc22e0bffccaea914efa13002d2c13b8a2ec
Open the JSON recordInspect the separate model-only skill testTry a local code check
The JSON record includes the case, policy, verifier, and output files.