Behavioral Eval guide

Behavioral Eval is the load-bearing differentiator for Skills. It runs your skill's trigger probes (requests that SHOULD activate it) and non-trigger probes (requests that should NOT) against every selected model in parallel, then scores each (probe × model) cell with an LLM judge. Anthropic's Agent Skills overview describes single-provider activation; PA's eval extends the lens to Claude + GPT + Gemini in one run.

Generate probes from your skill

Authoring probes by hand works, but the fastest way to get a starting set is to let the AI propose them. Click Generate from skill on the left rail; one BYOK inference call (your Workbench Model) returns 4-6 trigger probes (varied phrasings · short-form + long-form · different verbs and specificity) and 2-3 non-trigger probes (a mix of off-topic requests + one boundary-adjacent shape that tests over-firing).

If you have probes already authored, you'll get a Replace / Append / Cancel choice. Your work is never silently overwritten. Cost: a single small call against your Workbench Model (typically a few cents on Sonnet-class, less on Haiku / Nano / Flash). Probes never mention the skill's name (real users don't), so the generated set tests genuine discovery, not name-routing.

How to run one

1. 1
   Pick models

   Click the Behavioral Eval tab. Select 1-5 models across the providers you have BYOK keys for. Every tier shares the same 5-model cap; cross-provider testing requires a BYOK key per provider.
2. 2
   Author trigger probes

   These are user requests where the skill SHOULD activate. Write 4-12 of them. Click Generate from skill to have your Workbench Model propose a starting set from your SKILL.md (see Generate probes from your skill above).
3. 3
   Author non-trigger probes

   Requests that should leave the skill dormant. Without these, you can't measure false-positive activation. Every tier supports up to 6 non-trigger probes (and up to 10 trigger probes) per run.
4. 4
   Pick the judge model

   The cheapest available BYOK model is the default. The judge scores activation accuracy + adherence on each cell.
5. 5
   Run

   The grid streams live. Each cell shows the model's output, an activation indicator, and a 0-10 adherence score. Aggregate scores compute when all cells finish.

What the scores mean

Worked example · pdf-extractor

Let's walk through a real Behavioral Eval setup end-to-end. The skill we're scoring: pdf-extractor, a small capability that extracts plain text from a user-attached PDF and surfaces clause-level Q&A on the contents. The Skill should activate when the user mentions a PDF or asks about its contents, and stay dormant on every off-topic request.

Skill name

pdf-extractor

One-line description

Extracts plain text from an attached PDF and answers clause-level questions about its contents.

Trigger probes (4)

1. Extract text from this PDF
2. Summarize the contents of this PDF in 3 bullet points
3. What does the attached PDF say about clause 3?
4. Pull the dates and dollar amounts out of this contract PDF

Each probe phrases the same intent ("work with the PDF I've shared") differently. That distribution is the whole point: a model that fires on probe 1 but misses probe 4 has a discovery problem you can't catch with a single probe.

Non-trigger probes (3)

1. Write a haiku about elephants
2. What is the capital of France?
3. Help me debug this Python stack trace

These are distinctly off-topic. None of them mention PDFs, documents, or contracts. A pdf-extractor Skill that fires on "What is the capital of France?" is over-eager and will degrade the agent for every user who asks an unrelated question.

Models picked (3)

• Claude Opus 4.7 (Anthropic)
• GPT-5.5 (OpenAI)
• Gemini 2.5 Pro (Google)

Judge

Auto · cheapest BYOK. With keys for all three providers connected, the judge resolves to Haiku 4.5 or GPT-4.1 Nano (whichever is cheapest at run time).

Sample output

After running, the panel shows 12 cells (4 trigger × 3 models) plus 9 non-trigger cells (3 × 3) = 21 cells. Across the 12 trigger cells, all three models fire correctly on probes 1-3; only Gemini misses probe 4 (the contract-specific phrasing). Across the 9 non-trigger cells, Claude and GPT correctly stay dormant on all three; Gemini fires on "Help me debug this Python stack trace" because the word "trace" is loose enough to look like a document operation.

Discovery Fidelity

9.2 / 10 · 19 of 21 cells matched expected activation. The two misses are both Gemini.

Instruction Adherence

8.7 / 10 · when the Skill fired, the average judge score on instruction-following was high. Claude scored 9.1, GPT 8.8, Gemini 8.2.

Cross-Provider Consistency

7.4 / 10 · 10 minus the standard deviation of adherence across providers (0.45). Gemini's lower adherence pulls this down.

What to do with this

• Discovery is healthy (≥ 8.0) → no urgent rework. The two misses are concentrated in one provider.
• Adherence is healthy (≥ 8.0) → instructions are clear; the model that activated knew what to do.
• Consistency is the weakest score (7.4) → tighten the SKILL.md description field to make the contract case (probe 4) more discoverable on Gemini, and add a Do NOT activate for: example calling out generic "trace" requests so Gemini stays dormant.
• Re-run after the edit. If consistency stays under 8.0, mark the Skill as compatibility: [claude, openai] in the frontmatter and document the Gemini gap in the body.

Worked example · commit-message-formatter

A different domain to show range. Skill: commit-message-formatter, which turns a paste of git diff output or a request for a commit message into a Conventional Commit. The discovery surface is narrower (the user has to mention a diff, a commit, or paste recognizable diff text) but the adherence rubric is sharper (Conventional Commit format is rigid).

Trigger probes (5)

1. Write a commit message for these changes:\n\n``\ndiff --git a/lib/auth.ts ...\n``
2. Format this as a conventional commit: added null check in middleware
3. I just fixed a race condition in the session refresh, what should the commit message be?
4. Generate the commit subject + body for the staged changes
5. Turn this paragraph of release notes into a commit message

Non-trigger probes (3)

1. Review this PR description and make it sharper
2. Write a release note for v2.4
3. Explain what this Python decorator does

These deliberately sit close to the trigger surface. PR descriptions, release notes, and code explanations are adjacent to commit messages but distinct. A well-tuned Skill stays dormant on all three; an over-eager Skill blurs the boundary and writes commit-style output for a release note.

Sample output

Discovery Fidelity

8.5 / 10 · Claude and GPT fire correctly on all 5 trigger probes. Gemini misses probe 5 (the release-notes-into-commit case). Both Claude and GPT correctly stay dormant on all 3 non-trigger probes; Gemini fires on the "PR description" probe.

Instruction Adherence

7.8 / 10 · Conventional Commit format is rigid; even when the Skill fires, GPT occasionally drops the trailing blank line or exceeds the 72-char subject limit. Adherence per provider: Claude 8.6, GPT 7.4, Gemini 7.5.

Cross-Provider Consistency

8.8 / 10 · adherence is closer across providers than the pdf-extractor case.

How to write a good probe

Probes are user requests, written exactly as a real user would type them. The model only sees the probe text. It doesn't see the Skill name, the eval scaffold, or any meta-commentary. Five guidelines, with before/after examples:

1 · Be specific about user intent

• Bad · "Help me commit this" (too vague; could be code review, could be staging, could be a commit message)
• Good · "Format: feat(auth): add SSO" (the user is staging a specific commit message)

2 · Use natural language

Write how a real user would type, not how a test framework would phrase it. Casual lowercasing, partial sentences, missing punctuation are all fine; they're how users actually write.

• Bad · "Test case 1: invoke the pdf-extractor skill on a PDF input." (testing-language; will skew Discovery Fidelity)
• Good · "can you pull the dates out of this PDF?" (how a user actually asks)

3 · Don't lead the witness

Never include the Skill name in a probe. The model isn't supposed to know the Skill exists by name; it's supposed to discover the Skill from the request shape.

• Bad · "This should fire the pdf-extractor skill: extract text from foo.pdf" (telegraphs activation; trivializes Discovery Fidelity)
• Good · "extract the text from foo.pdf" (lets the model discover the Skill from the request)

4 · Distribute trigger phrasings

Don't write five probes that all start the same way. The point of multiple probes is to cover the space of how users phrase the same intent; if your probes are too similar, you'll miss real-world coverage gaps.

• Bad · all 5 probes start with "Extract text from…" (covers one phrasing five times)
• Good · mix of "Extract", "Summarize", "What does X say about Y", "Pull the X out of Y" (covers the actual surface)

5 · Pick distinctly off-topic non-triggers

Non-trigger probes test for false-positive activation. If your non-triggers are too close to the trigger surface, you're testing a different question (boundary discrimination, which is fine but harder). For a first eval, pick clearly off-topic non-triggers.

• Bad · for pdf-extractor, a non-trigger of "summarize this Word doc" (Word docs are document operations; boundary case, not off-topic)
• Good · for pdf-extractor, a non-trigger of "what is the capital of France?" (clearly unrelated; will surface false positives unambiguously)

Reading the score grid

Each cell in the per-cell results table represents one (probe × model) pair. The cell's background tint encodes the outcome:

Click any row to expand inline detail showing the full target output, judge reasoning, latency, token counts, and per-cell cost. The output is rendered as plain text · Prompt Assay never uses dangerouslySetInnerHTML on model output.

Diagnosis · why these scores

When a run completes with any aggregate score below 8.5, the workbench fires one extra workbench-model call that synthesizes the per-cell judge reasoning into 2-4 named failure themes plus a one-sentence summary. The card renders between the aggregate scores and the per-cell grid. Reads like a diff: "models stall at the env-check step and never produce the deliverable" or "non-trigger probes incorrectly activate when the model name appears in the request."

• Auto-fire: only when a score is below the 8.5 action threshold. All-strong runs render "every cell scored above the action threshold" and skip the LLM call entirely.
• One-shot: the diagnosis persists to the report row and never re-fires. Re-running the BE writes a fresh report with its own diagnosis.
• Severity tiering: each theme carries a High impact / Medium impact / Low impact chip. Themes with a target_dimension (Discovery / Adherence / Consistency / etc.) carry a second chip pointing to the Critique dimension they roll up to.
• Suggested actions are concrete: each theme's suggested action targets a specific edit rather than a generic recommendation, so you can act on it without re-reading every cell.

Improve from this eval

Below the diagnosis themes, an Improve from this eval button hands the synthesized themes plus the worst-cell evidence to the Improve panel. Click it, the workbench routes to the Improve tab, and the panel auto-fires Improve seeded with the BE context. Suggestions target the failure modes the eval surfaced rather than re-grading from scratch.

1. Run a Behavioral Eval and let the diagnosis settle.
2. Click Improve from this eval below the themes.
3. The workbench navigates to the Improve tab; the panel auto-fires with the BE diagnosis context loaded.
4. Apply the targeted suggestions like any Improve run · in-place edits + extract-to-file proposals are both available.
5. Save the new version, re-run the BE to verify the failure themes have resolved.

Cost guidance

Behavioral Eval bills your BYOK provider account directly. Prompt Assay never marks up. Cost scales linearly with cell count: each cell is one target call + one judge call.

Cell count formula

(trigger probes + non-trigger probes) × models

Total LLM calls

cell count × 2 (one target, one judge)

Quick math

• Per-run max (every tier) · 5 models × (10 + 6) probes = 80 cells × 2 = 160 LLM calls per run
• Typical 3-model run · 3 models × (6 + 4) probes = 30 cells × 2 = 60 LLM calls per run
• Single-provider sanity check · 1 model × (4 + 2) probes = 6 cells × 2 = 12 LLM calls per run

At Sonnet 4.6 pricing (~$3 per million input + $15 per million output), a typical 3-model run with 30 cells × ~500 input tokens × ~300 output tokens lands around $0.30-0.80. Max-config runs (5 models × 16 probes = 80 cells) with longer skills can push toward $3-5, well under the cost-preflight gate.

The $5 preflight gate

Before any run starts, the server estimates the cost from selected models, probe count, and rough token estimates derived from your SKILL.md size. Estimates under $5 run immediately. Estimates over $5 surface a confirmation dialog with the breakdown; you have to explicitly tap "Run anyway" to proceed. The per-run cap (80 cells = 160 LLM calls) stays bounded at the LLM-call layer regardless of model pricing.