Survey experiments

Experiments require @userreason/js 0.4.0 or later. They randomly assign people to defined survey branches. They are separate from answer-based routing and from shuffling choices.

Read Writing useful surveys before choosing what to compare. Define a hypothesis and comparable measures before reading the results. UserReason does not automatically name a winner.

Create an experiment

In the dashboard, use Create variant in a question's menu. The canvas shows an experiment node and its branches. Settings include the hypothesis, version names, shares, baseline, shared continuation, comparison mapping and an optional duration and sample-size plan. Preview can force a version without recording a real assignment or response.

The CLI and MCP save the same complete survey definition. content.flow is an explicit graph with version: 1, entry and nodes. Each question has one question node with its ID; experiment nodes choose a destination by weight. Every version must reach the experiment's shared join. A version may go straight to that join to test omitting its branch.

There can be 2–5 versions in one experiment and up to five experiments in a survey. Weights are positive integer basis points and total 10000, so 5000 means 50%. Graphs cannot loop, enter a version from outside its split, overlap branches before the join, or nest experiments. Sequential experiments can follow a completed join. An explicit graph replaces question-level logic routing; put conditions on its question nodes.

This complete definition compares two rating wordings and then asks a shared question:

{
  "schema_version": 1,
  "name": "Setup wording",
  "priority": 0,
  "sampling_percent": 100,
  "enrollment_epoch": "v1",
  "content": {
    "schema_version": 1,
    "title": "Setup feedback",
    "description": null,
    "thank_you": "Thank you for your feedback.",
    "questions": [
      { "id": "a", "kind": "rating", "prompt": "How clear was setup?", "required": true, "min": 1, "max": 5, "min_label": "Not at all clear", "max_label": "Very clear" },
      { "id": "b", "kind": "rating", "prompt": "How easy was it to understand the setup instructions?", "required": true, "min": 1, "max": 5, "min_label": "Very difficult", "max_label": "Very easy" },
      { "id": "after", "kind": "text", "prompt": "What, if anything, could we improve?", "required": false, "max_length": 2000 }
    ],
    "flow": {
      "version": 1,
      "entry": { "type": "experiment", "id": "wording" },
      "nodes": [
        {
          "type": "experiment", "id": "wording", "name": "Setup wording",
          "hypothesis": "Question wording may change how people rate setup.",
          "control_variant_id": "control",
          "join": { "type": "question", "id": "after" },
          "variants": [
            { "id": "control", "name": "Clarity", "weight": 5000, "to": { "type": "question", "id": "a" } },
            { "id": "alternative", "name": "Understanding", "weight": 5000, "to": { "type": "question", "id": "b" } }
          ],
          "comparisons": [
            { "id": "setup-rating", "name": "Setup rating", "kind": "rating", "questions": { "control": "a", "alternative": "b" } }
          ],
          "analysis": { "metric": "answer_rate", "duration_days": 7, "minimum_subjects": 100 }
        },
        { "type": "question", "id": "a", "otherwise": { "type": "question", "id": "after" } },
        { "type": "question", "id": "b", "otherwise": { "type": "question", "id": "after" } },
        { "type": "question", "id": "after", "otherwise": { "type": "end" } }
      ]
    }
  },
  "rules": {
    "schema_version": 1,
    "audience": { "type": "always" },
    "triggers": [{ "type": "page_load" }],
    "frequency": { "mode": "site_default" }
  }
}

The example's duration and sample size are illustrative, not a power calculation or a stopping guarantee. Its scales share numeric bounds, but wording still changes what they ask. Judge whether comparing them answers your intended question.

Comparison questions maps variant IDs to question IDs in their branches. kind: "rating" compares numeric ratings; equal bounds are necessary for a shared scale, not proof of equivalent meaning. kind: "distribution" keeps answers separate. For kind: "categorical", supply categories: [{id, label}] and options: {question_id: {option_id: category_id}}; map every compared choice to an existing category. Match by meaning, not by where the option appears.

Assignment and publication

The backend assigns each person one version per experiment run. Later visits in that run retain it. Identifying an anonymous visitor can reveal an existing assignment; identity conflicts remain visible in reporting rather than being silently rewritten out of history.

Every publication starts new runs for all experiments in the published definition, even when only other content changed. Previous runs keep their pinned definitions and results. Pausing stops new invitations; people who already started can finish. Resuming retains the current runs and assignments. Moving canvas cards alone is editor metadata and is not a content change to publish.

The compatible SDK requests presentation=v1. Older SDKs are not sent surveys requiring it. Confirm the SDK before declaring presentation:v1 in validation. Keep enrollment_epoch unchanged on ordinary edits; a new experiment run and a new survey enrollment are different concepts.

Validate and save

With @userreason/cli 0.1.1 or later installed, run from a project whose SDK target is configured, or supply --workspace and --site explicitly:

userreason prompt validate --file experiment.json --context-file sdk-capabilities.json --json
userreason prompt create --file experiment.json --json

{ "client_capabilities": ["presentation:v1"] }

For an existing survey, read it with prompt get and update the full definition using its current --revision. Saving is separate from publishing. Read the returned links.edit to review the draft, then publish the approved survey using the current revision. See CLI commands.

Validation preserves issues, warnings, valid, ready and status, and also returns:

These checks do not measure question neutrality or sample representativeness. Validation saves nothing and does not create visitor history.

Read results

The dashboard Experiments tab and the CLI/MCP report use the same backend calculations:

userreason results experiments "$PROMPT_ID" --json
userreason results experiments "$PROMPT_ID" --run "$RUN_ID" --reported-only --json
userreason results experiments "$PROMPT_ID" --run "$RUN_ID" --all-attempts --json
userreason responses list --prompt-id "$PROMPT_ID" --run "$RUN_ID" --variant control --json

PROMPT_ID is the survey UUID returned by creation, and RUN_ID comes from the report's runs. Without --run, the report selects the latest available run. No experiment is required to retrieve a survey's answer-position data.

MCP exposes results_experiments with workspace_id, site_id, prompt_id, optional run_id, all_attempts and reported_only. responses_list accepts optional run_id and variant_id; a variant requires a run. The report returns links.experiments, while response links retain the run and variant filters. These reads require site:read and current workspace access.

Default comparisons count the first reached attempt per person and run. all_attempts is descriptive: repeat visits are not independent participants. reported_only excludes inferred split entries; it does not mean every order was confirmed. Position statistics separately track reported and order-confirmed question presentations, across the pinned survey version.

Answer and completion rates use reached as the denominator. They are conditional rates, not an automatic estimate of the effect on every allocated person. Inspect allocation, reach, dropout, answer distributions, identity crossovers and sample size together. The 95% rate intervals are not a probability that a version won; overlap or lack of overlap alone is not a significance test. The plan is recorded context, not an automatic stopping rule.

Historical responses keep their original question labels, variant and saved order. Use responses list --json for experiments, presented_questions, presentation_orders and presentation_evidence. See answer-order evidence before treating a saved order as proof of exposure.