Writing useful surveys

Start with the decision you want the answers to inform. Ask only what you need to make that decision, from people who have had the relevant experience. A technically valid survey can still ask the wrong question at the wrong moment.

Make participation clearly optional

A survey shown during a task can look like a required step. Make the invitation clear, let people dismiss it, and avoid implying that answering unlocks the product unless that is actually the purpose. Prefer a natural pause after an experience over interrupting an unfinished task.

Do not preselect an answer on the person's behalf. A first answer selected to move past a prompt is not necessarily an opinion. Consider what a high dismissal rate or unusually fast completion could mean before interpreting the answers.

Ask one neutral question at a time

Use words the respondent understands, specify the experience or time period, and separate questions that ask about different things.

Give balanced, distinct answer options. Include “Not applicable”, “Don't know” or an optional question when a person may reasonably have no answer. Use an explicit own-words option when the listed choices may be incomplete. Keep meaning consistent across translations.

Match answer order to the question

For an unordered list of reasons, answer randomisation can distribute which option appears first. Keep a meaningful sequence, such as a rating scale, chronology, or a list people expect to scan alphabetically, in its natural order. Pin options such as “Other” at the end where appropriate.

Randomisation addresses position effects. It does not repair leading language, missing options, an unsuitable audience or pressure to respond. Earlier questions may also influence later answers; shuffling answer options does not shuffle questions.

Use an experiment for a specific uncertainty

Use an experiment when you want to compare defined versions of a question or a short question path. Write down what you expect to learn, which answers are comparable, and how long and how many people you intend to observe before publishing.

Keep differences deliberate. If wording, options and timing all change together, the result cannot tell you which change mattered. A higher response rate or a friendlier answer does not by itself show that a version measures the experience more accurately.

Read responses in context

Look at who was eligible, who was shown the survey, who answered and who left. People who respond may differ from people who dismiss it or never encounter it. Treat survey feedback as evidence about the people and moment sampled, rather than an automatic estimate of every customer's opinion.

Read counts alongside percentages. Inspect written answers and partial responses when relevant, and compare equivalent time periods and populations. Small samples can produce large apparent differences. Report uncertainty and limitations rather than treating the largest number as a winner.

What validation checks

Validation checks concrete configuration: field shapes, graph connections, choice pins, experiment shares, publication requirements and declared SDK capabilities. It cannot establish that the wording is neutral or that the sample represents your users.

The dashboard, CLI and MCP share server-side rules. Validation returns concise diagnostics with a code, field path, severity and documentation link. Keep methodological guidance here in the docs; use the diagnostics to find and fix the particular configuration issue.