Prompt Assay
§.Migrate from Promptfoo

YAML in.
Workbench out.

Promptfoo’s configuration is YAML. Parse `promptfooconfig.yaml`, map prompts and tests into Prompt Assay’s library and eval surface, and land in a workbench with no parent-company tilt and no per-trace tax.

I.The procedure

Three steps. Then you ship.

  1. Station · 01

    Parse

    Your existing `promptfooconfig.yaml` is the migration source of truth. Parse the file with PyYAML or any YAML library; the structure is well-defined. The fields you care about: `prompts` (the prompt strings or files), `providers` (the model configs), `tests` (the test cases with `vars` and `assert` blocks), and `redteam` (if you used the security probe sets). Keep the YAML around · the parse is non-destructive.

  2. Station · 02

    Map

    Each `prompts` entry becomes a new prompt in Prompt Assay's library; preserve variable interpolation as `{{var}}` placeholders. Each `tests` entry becomes a test case in an eval suite, with the `assert` blocks mapping to graders (rubric-based or LLM-as-a-judge depending on the assertion shape). `providers` configs become run-time provider selections; BYOK keys connect directly to Anthropic, OpenAI, Google with no parent-company tilt. `redteam` configurations are not Prompt Assay's lane · pair with a security-testing surface for that workflow.

  3. Station · 03

    Land

    Sign up for Prompt Assay. Create a prompt for each parsed entry; paste the body in as version one and re-add variables. Build the eval suite, attach test cases and graders, and run it. The AI pair (Brainstorm, Critique, Improve, Rewrite, Compare) is available from the first save; six-dimension critique runs on every prompt before it ships.

II.The destinations

Four rows. Five facts each.

Each row is a real option. Pick the row that matches your eval-CI shape and your sensitivity to OpenAI's roadmap stewardship.

  • Promptfoo

    Platform fee
    Open-source CLI free (MIT-licensed). Enterprise tier custom; pricing not publicly disclosed post-acquisition. Acquired by OpenAI on Mar 9 2026; deal value not disclosed. OpenAI committed to 'continue building out' the open-source offering.
    Provider scope
    Multi-provider via YAML-defined providers in `promptfooconfig.yaml`. Historical scope: 250+ providers including Anthropic, OpenAI, Google, Bedrock, local models. Future non-OpenAI provider commitment is not publicly stated.
    Inference path
    Direct to provider. CLI runs locally; provider keys never leave your machine.
    Promptfoo import
    YAML-driven test cases. Prompt strings, eval assertions, and provider configs all in `promptfooconfig.yaml`. Migration is parsing the YAML and re-creating the prompt + eval suite in the new tool.
    Best fit
    AI security and red-teaming workflows that integrate into OpenAI Frontier post-acquisition. Multi-provider eval users now subject to OpenAI's roadmap.

  • Braintrust

    Platform fee
    Starter free (1GB processed, 10K scores). Pro $249/mo (5GB, 50K scores, 30-day retention). Enterprise custom. Closed-source, hosted-only on Pro; self-host is Enterprise-only.
    Provider scope
    Multi-provider via AI Gateway accepting OpenAI, Anthropic, and Google SDKs.
    Inference path
    BYOK supported, but inference flows through Braintrust's gateway. Provider keys still pass through their proxy.
    Promptfoo import
    Prompts are TypeScript-defined. Promptfoo migration requires manual transformation: parse YAML, re-author prompts in TS, port assertions to Braintrust's eval format.
    Best fit
    Well-funded eval-driven teams who want a managed gateway plus eval CI surface and accept proxied inference.

  • Langfuse

    Platform fee
    Hobby free (50K units/mo) · Core $29/mo · Pro $199/mo · Enterprise $2,499/mo. Self-host free under MIT.
    Provider scope
    Provider-neutral via OpenTelemetry. Multi-provider tracing and prompt management.
    Inference path
    Direct to provider. OTel instrumentation; no proxy in the inference path.
    Promptfoo import
    Datasets API accepts re-imported test cases. Promptfoo YAML must be parsed and pushed via Public API.
    Best fit
    Open-source-first teams who want OTel-native tracing alongside dataset and eval primitives, with self-host as a clean compliance fallback.
  • Our entry

    Prompt Assay

    Platform fee
    Free tier · Solo $49/mo · Team $99/seat/mo · Enterprise contact-sales. BYOK-mandatory at every paid tier with no inference markup.
    Provider scope
    Anthropic, OpenAI, Google with first-class adapters and no parent-company tilt.
    Inference path
    Direct to provider. We never sit in the inference request path. Your bill stays with your provider account.
    Promptfoo import
    No first-class YAML parser. Copy each Promptfoo prompt into a new Prompt Assay prompt; eval suites land in the workbench's evaluation surface (test cases + rubrics + LLM-as-a-judge).
    Best fit
    Multi-provider teams who picked Promptfoo for OSS neutrality and want a workbench that pairs authoring (six-dimension critique, two-version Compare) with the eval surface in a single tool, with no parent-company provider preference.

Verified 2026-05-01 · Read the full breakdown

III.Marginalia · 5 questions

Frequently asked.

Is the Promptfoo OSS license going away after the OpenAI acquisition?
No. The MIT license is preserved per OpenAI's announcement, and OpenAI stated it 'expects to continue building out Promptfoo's open source offering.' The forward-looking framing matters · OpenAI has not committed publicly to maintaining feature parity for non-OpenAI providers (Anthropic, Google, Bedrock, local models) going forward, and the integration target is OpenAI Frontier. The migration question is roadmap stewardship, not platform death.
Do I have to migrate? Can I keep using Promptfoo alongside Prompt Assay?
You don't have to. Many teams already run both today · YAML-defined eval CI in Promptfoo for pre-deploy checks, plus a hosted tool for production observability and authoring. Adding Prompt Assay alongside is a low-cost first step: keep Promptfoo for what it does well, and add the workbench surface upstream for authoring, six-dimension critique, two-version Compare, and prompt-level versioning. The decision to fully retire Promptfoo can come later when the roadmap signal is clearer.
What about the redteam configurations?
Red-teaming and AI-agent security probing are Promptfoo's strongest surface, and they are not Prompt Assay's lane. If your primary use was red-teaming, the migration calculus is different · pair Prompt Assay (or any other workbench) with a dedicated security-testing tool for the red-team surface. OpenAI Frontier is one option post-acquisition; there are independent security-research tools that occupy this space too.
Will my assertion library carry over?
The shape carries over; the syntax does not. Promptfoo's `assert` blocks (icontains, javascript, llm-rubric, model-graded-closedqa) map to Prompt Assay eval-suite graders · rubric checks land as scoring rubrics, LLM-as-a-judge checks land as judge-model graders, deterministic string checks land as expected-output assertions on the test case. The mapping is one-to-one in concept; the YAML-to-UI configuration step is what takes the time.
Why pick Prompt Assay over Braintrust or Langfuse for this migration?
Three honest reasons. Authoring depth · Prompt Assay covers the prompt-iteration loop upstream of evals (six-dimension critique, two-version Compare with model-graded diff, AI pair) where Braintrust and Langfuse are eval-and-tracing-first. Provider neutrality · BYOK at every paid tier with no inference markup or proxy in the request path; Braintrust runs a managed gateway, Langfuse is OTel-native but optimizes for self-host or hosted at higher tiers. Pricing shape · Prompt Assay Team is flat at $99 per seat per month with no per-trace tax, no per-GB-processed billing, and no per-score overage.
IV · Closing

Land in the workbench.

Free to start. Your keys, your bill, no parent-company tilt.

Start your migrationRead the full breakdown →