---
title: Survey experiments
description: Create weighted survey branches, keep variant assignment stable, validate publication, and read experiment results through the dashboard, CLI or MCP.
url: https://docs.userreason.com/prompts/experiments/
updated: 2026-10-17
---

# Survey experiments

Experiments require `@userreason/js` 0.4.0 or later. They randomly assign people to defined survey branches. They are separate from answer-based routing and from [shuffling choices](https://docs.userreason.com/prompts/answer-order/index.md).

Read [Writing useful surveys](https://docs.userreason.com/prompts/survey-quality/index.md) before choosing what to compare. Define a hypothesis and comparable measures before reading the results. UserReason does not automatically name a winner.

## Create an experiment

In the dashboard, use **Create variant** in a question's menu. The canvas shows an experiment node and its branches. Settings include the hypothesis, version names, shares, baseline, shared continuation, comparison mapping and an optional duration and sample-size plan. Preview can force a version without recording a real assignment or response.

The CLI and MCP save the same complete survey definition. `content.flow` is an explicit graph with `version: 1`, `entry` and `nodes`. Each question has one question node with its ID; experiment nodes choose a destination by weight. Every version must reach the experiment's shared `join`. A version may go straight to that join to test omitting its branch.

There can be 2–5 versions in one experiment and up to five experiments in a survey. Weights are positive integer basis points and total **10000**, so 5000 means 50%. Graphs cannot loop, enter a version from outside its split, overlap branches before the join, or nest experiments. Sequential experiments can follow a completed join. An explicit graph replaces question-level `logic` routing; put conditions on its question nodes.

This complete definition compares two rating wordings and then asks a shared question:

```json title="experiment.json"
{
  "schema_version": 1,
  "name": "Setup wording",
  "priority": 0,
  "sampling_percent": 100,
  "enrollment_epoch": "v1",
  "content": {
    "schema_version": 1,
    "title": "Setup feedback",
    "description": null,
    "thank_you": "Thank you for your feedback.",
    "questions": [
      { "id": "a", "kind": "rating", "prompt": "How clear was setup?", "required": true, "min": 1, "max": 5, "min_label": "Not at all clear", "max_label": "Very clear" },
      { "id": "b", "kind": "rating", "prompt": "How easy was it to understand the setup instructions?", "required": true, "min": 1, "max": 5, "min_label": "Very difficult", "max_label": "Very easy" },
      { "id": "after", "kind": "text", "prompt": "What, if anything, could we improve?", "required": false, "max_length": 2000 }
    ],
    "flow": {
      "version": 1,
      "entry": { "type": "experiment", "id": "wording" },
      "nodes": [
        {
          "type": "experiment", "id": "wording", "name": "Setup wording",
          "hypothesis": "Question wording may change how people rate setup.",
          "control_variant_id": "control",
          "join": { "type": "question", "id": "after" },
          "variants": [
            { "id": "control", "name": "Clarity", "weight": 5000, "to": { "type": "question", "id": "a" } },
            { "id": "alternative", "name": "Understanding", "weight": 5000, "to": { "type": "question", "id": "b" } }
          ],
          "comparisons": [
            { "id": "setup-rating", "name": "Setup rating", "kind": "rating", "questions": { "control": "a", "alternative": "b" } }
          ],
          "analysis": { "metric": "answer_rate", "duration_days": 7, "minimum_subjects": 100 }
        },
        { "type": "question", "id": "a", "otherwise": { "type": "question", "id": "after" } },
        { "type": "question", "id": "b", "otherwise": { "type": "question", "id": "after" } },
        { "type": "question", "id": "after", "otherwise": { "type": "end" } }
      ]
    }
  },
  "rules": {
    "schema_version": 1,
    "audience": { "type": "always" },
    "triggers": [{ "type": "page_load" }],
    "frequency": { "mode": "site_default" }
  }
}
```

The example's duration and sample size are illustrative, not a power calculation or a stopping guarantee. Its scales share numeric bounds, but wording still changes what they ask. Judge whether comparing them answers your intended question.

Comparison `questions` maps variant IDs to question IDs in their branches. `kind: "rating"` compares numeric ratings; equal bounds are necessary for a shared scale, not proof of equivalent meaning. `kind: "distribution"` keeps answers separate. For `kind: "categorical"`, supply `categories: [{id, label}]` and `options: {question_id: {option_id: category_id}}`; map every compared choice to an existing category. Match by meaning, not by where the option appears.

## Assignment and publication

The backend assigns each person one version per experiment run. Later visits in that run retain it. Identifying an anonymous visitor can reveal an existing assignment; identity conflicts remain visible in reporting rather than being silently rewritten out of history.

Every publication starts new runs for all experiments in the published definition, even when only other content changed. Previous runs keep their pinned definitions and results. Pausing stops new invitations; people who already started can finish. Resuming retains the current runs and assignments. Moving canvas cards alone is editor metadata and is not a content change to publish.

The compatible SDK requests `presentation=v1`. Older SDKs are not sent surveys requiring it. Confirm the SDK before declaring `presentation:v1` in validation. Keep `enrollment_epoch` unchanged on ordinary edits; a new experiment run and a new survey enrollment are different concepts.

## Validate and save

With `@userreason/cli` 0.1.1 or later installed, run from a project whose SDK target is configured, or supply `--workspace` and `--site` explicitly:

```sh
userreason prompt validate --file experiment.json --context-file sdk-capabilities.json --json
userreason prompt create --file experiment.json --json
```

```json title="sdk-capabilities.json"
{ "client_capabilities": ["presentation:v1"] }
```

For an existing survey, read it with `prompt get` and update the full definition using its current `--revision`. Saving is separate from publishing. Read the returned `links.edit` to review the draft, then publish the approved survey using the current revision. See [CLI commands](https://docs.userreason.com/cli/index.md#surveys).

Validation preserves `issues`, `warnings`, `valid`, `ready` and `status`, and also returns:

| Field | Meaning |
| - | - |
| `diagnostics` | Concrete issue code, field path, affected `node_id` when available, severity, short message, `blocks_publication` and `documentation_url`. |
| `required_capabilities` | Capabilities this definition needs. An unconfirmed capability can make the rehearsal `unsupported_client` without making the definition invalid. |
| `publication.new_experiment_runs` | The experiments and shares that publishing this definition would start. This is a preview, not a publication receipt. |
| `links.documentation` | Relevant documentation in the selected API environment. |

These checks do not measure question neutrality or sample representativeness. Validation saves nothing and does not create visitor history.

## Read results

The dashboard Experiments tab and the CLI/MCP report use the same backend calculations:

```sh
userreason results experiments "$PROMPT_ID" --json
userreason results experiments "$PROMPT_ID" --run "$RUN_ID" --reported-only --json
userreason results experiments "$PROMPT_ID" --run "$RUN_ID" --all-attempts --json
userreason responses list --prompt-id "$PROMPT_ID" --run "$RUN_ID" --variant control --json
```

`PROMPT_ID` is the survey UUID returned by creation, and `RUN_ID` comes from the report's `runs`. Without `--run`, the report selects the latest available run. No experiment is required to retrieve a survey's answer-position data.

MCP exposes `results_experiments` with `workspace_id`, `site_id`, `prompt_id`, optional `run_id`, `all_attempts` and `reported_only`. `responses_list` accepts optional `run_id` and `variant_id`; a variant requires a run. The report returns `links.experiments`, while response links retain the run and variant filters. These reads require `site:read` and current workspace access.

| Count | What it establishes |
| - | - |
| Allocated | A person received a version, including people who never reached the split. |
| Reached | Entry reported by the browser or inferred from accepted branch answers. |
| Reported | The browser explicitly reported reaching the split. |
| Displayed | The browser reported a version question being shown. |
| Answered / completed | A saved answer to the compared question / a completed survey after reaching the split. |

Default comparisons count the first reached attempt per person and run. `all_attempts` is descriptive: repeat visits are not independent participants. `reported_only` excludes inferred split entries; it does not mean every order was confirmed. Position statistics separately track reported and order-confirmed question presentations, across the pinned survey version.

Answer and completion rates use **reached** as the denominator. They are conditional rates, not an automatic estimate of the effect on every allocated person. Inspect allocation, reach, dropout, answer distributions, identity crossovers and sample size together. The 95% rate intervals are not a probability that a version won; overlap or lack of overlap alone is not a significance test. The plan is recorded context, not an automatic stopping rule.

Historical responses keep their original question labels, variant and saved order. Use `responses list --json` for `experiments`, `presented_questions`, `presentation_orders` and `presentation_evidence`. See [answer-order evidence](https://docs.userreason.com/prompts/answer-order/index.md#reading-the-results) before treating a saved order as proof of exposure.