Test and publish with quality
Test cases, suites, evaluations, and the quality gate before publishing.
Quality tests answer one question: does this version do what I expect? They run on saved versions, never on the draft.
The pieces#
| Piece | What it is |
|---|---|
| Test dataset | 1 to 200 cases. Each has a name, the customer's message, and optionally the expected answer |
| Suite | A dataset, the evaluators that grade each answer, and the share of cases that must pass |
| Evaluation | One run of the suite against an agent version |
| Quality gate | A suite every version must pass before it can be published |
Available evaluators#
| Evaluator | Passes when the answer… |
|---|---|
equals | is exactly the expected one |
contains | contains a text |
starts_with | starts with a text |
tool_called | used a specific tool |
output_rubric | meets a rubric, graded by a judge model |
1. Create the cases#
"Create test cases for 'Store support': a customer asks about their shipment without giving the order, another asks for a refund, and another says hello."
The chatbot proposes the cases and saves them with create_eval_dataset.
2. Build the suite#
"Build a suite with those cases. It passes if the agent asks for the order number when needed and never promises a refund. Require 100%."
For rules like "never promises a refund" the chatbot uses output_rubric, which a judge model grades.
3. Run the evaluation#
"Evaluate version 1 with that suite and tell me the result."
The evaluation runs in the background. The chatbot checks get_evaluation until it finishes and shows you each case with its grade.
Note
Each case is a real conversation with the model. That's why evaluations spend credits, and so does the judge model.
4. Require quality before publishing (optional)#
"Don't let any version of 'Store support' be published without passing this suite."
With set_quality_gate, publishing a version that didn't pass fails with quality_gate_not_met.
5. Publish#
"Publish version 1."
publish_agent_version changes what your customers get on every channel. That's why the chatbot is instructed to confirm with you first.
Important
Publishing affects your customers right away. If something goes wrong, publish the previous version: they are all still saved.