Maintenance
Part of Offer testing: climbing the commitment ladder one slot at a time
Most offer testing mistakes happen before the first person answers
Offer testing mistakes made before, during and after a test, with the one-page record that catches most of them before the first person is approached.
An offer test goes wrong at one of three moments: while it is being designed, while it is running, or while the result is being read. The errors at each stage look different in your data, and they need different repairs, so they are grouped that way here rather than ranked.
Nine in total. For each, the trace it leaves and the correction.
What to take away
- Design errors produce a result that cannot be interpreted. Running errors produce a result about the wrong thing. Reading errors produce the wrong conclusion from a good result.
- The cheapest fix for most of them is a sentence written down before the test starts.
- A test that cannot fail is not a test, and most of these mistakes are ways of quietly making failure impossible.
Before the test starts
1. Changing every slot at once
The trace. Version two of the page has a new headline, a new price and a new audience. It did better, or worse, and you cannot say why.
The correction. Write the offer as the five-slot sentence from the offer pillar, underline the one slot you are least sure of, and change only that. The other four hold still until this reading is in.
2. No sealed prediction
The trace. After the test, you find yourself explaining why the result is actually encouraging.
The correction. Before it runs, write one sentence: "I expect at least N of the M people who see this to do X." Put it somewhere you cannot edit quietly. Whatever the numbers turn out to be, the surprise between the prediction and the result is the finding.
3. Recruiting the wrong audience and calling it a test
The trace. The people who saw the offer are your contacts, your community, or whoever an advertising platform decided to show it to. The who slot says something else.
The correction. Before you count anything, describe who actually arrived and compare that with who the offer names. If they differ, the result is about the channel and says nothing about the offer. A warm audience is useful for one thing: checking the page works before strangers see it.
While the test runs
4. Asking for the wrong thing
The trace. You collected email addresses when the question was whether anyone would pay.
The correction. Choose the rung of the commitment ladder that matches the assumption at risk. If the risk is demand, ask for a refundable deposit or a signed order. If the risk is message, a scheduled call is enough. Asking for less than the question needs gives you a confident answer to a different question.
5. Adjusting mid-run
The trace. The price changed on day three because the first two days were quiet.
The correction. Set the run length in advance and hold it. Quiet early days are the normal condition of any small test. If you must change something, stop the test, record the result so far as a separate reading, and start a new one with a new prediction.
6. Counting interest as intent
The trace. The result includes survey answers, comments, and people who said they would definitely buy.
The correction. Count only the behavior you named in the prediction. Everything else goes in a notes column. It may be useful later for wording, and it is worth nothing as evidence of demand, because saying yes to a hypothetical costs the speaker nothing. How much the wording of a question shapes the answer is documented at length in the guidance on writing survey questions, and a survey you wrote about your own idea is the hardest case of all.
After the test ends
7. Reading a small count as a rate
The trace. "Two out of thirty" has become a percentage in your summary.
The correction. Report counts, not rates, until the counts are large enough that one more person would not change the picture. At the sizes most founders test at, two people is two people. Ask both of them why they acted. Their reasons carry more information than the ratio.
8. Blaming the offer for a delivery failure
The trace. The page had a broken form on mobile, or the traffic arrived on a public holiday, and the conclusion in your notes is that nobody wants this.
The correction. Before any conclusion about demand, rule out execution. Run the same offer through a direct conversation with five of the right people. If it converts there and not on the page, the offer is fine and the page or the channel is not. This one deserves its own line in the launch plan, because a public launch is the most expensive place to make it.
9. Taking money without a date
The trace. Pre-orders sit in an account with no delivery date attached, and the customers have stopped writing.
The correction. Every payment taken during a test carries a stated delivery date and a refund on request if you miss it. That is not a courtesy: the order and shipment rule requires notice, consent, or a prompt refund from a seller who cannot ship within the window stated. This is not only fair; it is what makes the payment evidence. A person who pays knowing exactly what state the thing is in has given you a stronger reading than one who assumed it was finished. The limits are set out under minimum viable offers: under-describe your machinery, never misdescribe your result.
The one-page defense
Most of the nine are prevented by a single sheet written before the test. It holds the offer sentence with one slot underlined, the audience described so you can check who arrived, the behavior being counted, the prediction, the run length, and the two thresholds that mean continue and stop. Six lines. Read it again before you interpret anything.
Common questions
What if the prediction is wildly wrong?
Good. A badly wrong prediction means the test taught you something you did not know. The failure mode is a prediction so vague it cannot be wrong.
Can I run two offers at once to save time?
To different audiences, with separate predictions, yes. To the same audience, no; they contaminate each other, and the price test in particular needs to be one number said plainly.
How do I know if a result was about the traffic and not the offer?
Describe who arrived. If you cannot, the test was not designed to be read, and the next run needs a channel where you can see who came.
Is it a mistake to test with a page at all?
No, but a page can only test the top of the commitment ladder. Once the message works, the next test is a conversation in which somebody can refuse a number.
