Notion authoring edition · 0.1.23. Portable distribution. Source review is complete; native Lab adoption remains unqualified.

Use as a shared evaluation method across software, BI, defence, research and other work. AgentOps commissions a bounded evaluation when its evidence can change a decision; the work module supplies domain criteria. An evaluator is a bounded worker, not a higher permission tier.

Start with a decision and explicit hypothesis. Select representative authorized cases and the evidence needed to distinguish alternatives. Fix the Factory declaration and selected skill revisions where used, input revisions, dependency and environment revisions, task criteria, harness/model versions, tools and relevant settings for comparison. Use enough repetitions for the claim; explain uncertainty rather than inventing a universal sample count or treating one example as a benchmark.

For a model-only comparison, start independent attempts from the same frozen native session/thread checkpoint. Retain its input history, declaration/skill versions, candidate files or data, tools, permissions and agreed budget; change only the model for the role under test and keep other roles fixed. Keep attempts from reading each other's outcomes. A transcript alone does not restore task or provider state. Record native lineage and unavoidable context differences; unavailable exact replay limits the conclusion. Subsequent answers and tool paths may diverge. Use the existing native fork or a bounded retained context package, not a custom replay engine.

Retain failures, timeouts, review/fix cycles and selection costs. Keep private expected results with the evaluator, separate from implementation. A worker must not modify its own oracle or acceptance criteria. Evaluate resulting artifacts and behavior independently of the model's self-rating.

Use objective domain checks where they exist. An optional model judge applies a predeclared rubric; its score is a judgment, not proof of correctness. Multiple judges need a purpose, disagreement handling and a budget. Review of a concrete candidate and cross-run evaluation are distinct activities. Count the lead, work, review, repair and evaluation in the attempt.

Report accepted outcomes, defects/rework, human intervention/time, elapsed time and available usage/cost. Unknown is null, not zero. Distinguish estimates, subscription windows and actual billed cost. For cost per accepted outcome, divide all attempt costs, including failed attempts, by accepted outcomes. Unknown material components leave the total unknown: report a lower bound or an inconclusive comparison, not a complete price claim. Token throughput alone is not value.

Return a report with an adopt, reject, inconclusive or defer recommendation for the specified task/configuration. Adoption and delivery follow existing authority. A PR is appropriate for a proposed method or configuration change; an experiment need not change product code.

Compare matched cases and disclose confounders, coverage and limits. A synthetic rehearsal establishes only that rehearsal's behavior; live installation and business outcomes require their own observations. Recommend the smallest useful change and its next check. Evaluation does not automatically adopt a new model, skill, account or spend policy.