Evals Are Deploy Infrastructure, Not a Research Sidecar
Teams gate deploys on tests and types, then ship prompt changes on vibes. Run rubric-based eval suites, benchmark agents on cost per successful task, and block any deploy that regresses — three pay-per-call skills, with a direct import path for your existing OpenAI Evals YAML.
Run → Benchmark → Gate
Three questions every production agent pipeline has to answer, each mapped to a skill — priced per call, paid in USDC via x402, no platform to adopt.
Is the output good?
Rubric-based LLM-judge suites with position-debiased pairwise scoring and 3-pass self-consistency, golden-set matching, and programmatic assertions — one call per suite run, with failure clustering by root cause.
Versioned suite documents; imports OpenAI Evals registry YAML directly, so existing suites run unchanged.
Is the agent worth what it costs?
HAL-style cost-aware benchmarking: success rate, cost per successful task computed from LLM spend plus x402 receipts, success-vs-cost Pareto curves by difficulty tier, and A/B comparison of 2–5 configurations with significance testing.
Reproducibility manifests pin versions, seeds, and temperature so a benchmark result means the same thing next month.
Did this change make it worse?
Eval-as-deploy-gate: register a baseline, gate every candidate with PASS/FAIL/WARN verdicts, per-dimension tolerance overrides (safety never drops), and run-to-run variance noise guards so gates fail on regressions, not noise.
Signed gate records for audit; composes with skill version history (gate before bumping a version) and DeployGuard canaries (gate before widening traffic).
Migrating from the OpenAI Evals Platform
The Evals Platform and Agent Builder shut down November 30, 2026. If your eval suites live there, this is the three-step landing pad — your registry YAML is the migration artifact, not a starting point for a rewrite.
Export your existing suites
Your OpenAI Evals registry YAML files — the suite definitions, golden sets, and judge configurations you already maintain — are the input. No rewriting into a new DSL.
Import and run
POST the YAML to eval-suite-runner. Suites become versioned BluePages suite documents and run with position-debiased LLM-judge scoring and self-consistency passes on the first call.
Promote a baseline and gate your deploys
Promote a known-good run to a baseline in regression-eval-gate, then call the gate from CI or a composition ahead of every prompt change, model swap, or version bump.
What changes in the move: judge scoring gains position debiasing and 3-pass self-consistency; suite runs gain failure clustering; and the gate primitive — which the Evals Platform never had — connects your suites to deploys instead of dashboards.
Why Skills Beat an Eval Platform
No platform to lose
The teams migrating right now bet on a platform that is being turned off. Skills are API calls against versioned suite documents you own — the suite definition travels with you.
Gates, not dashboards
Dashboards get looked at after the incident. A regression gate sits in CI or a composition and returns PASS/FAIL/WARN before the deploy — with a signed record that the version you shipped passed the evals you claimed.
Cost is a first-class metric
Benchmarks compute cost per successful task from LLM spend plus real x402 payment receipts — success-vs-cost Pareto curves instead of accuracy numbers that ignore what the accuracy costs.
Building Evaluation Infrastructure?
Domain-specific judge rubrics, red-team suites, human-preference collection, eval-data generation — the Evals Platform shutdown creates demand for evaluation skills that don't exist yet. List yours while displaced teams are choosing their next harness.