Offer testing metrics: fields to record before and after

One page per offer test, written in two sittings. The denominators and the decision rule go down before the test runs. The counts, the refusals and the drift go down after it. Written that way, the page still means something a month later.

An offer test produces one number that everybody looks at and four that decide whether it means anything. The four are denominators, and getting them wrong is the reason two tests of the same offer can disagree completely.

What to take away

The record, field by field

Each field belongs to one sitting. Fill the first group before anyone is contacted and the second after the test closes. Nothing in the first group changes once the test starts.

First sitting: the four denominators

Every offer test has four populations, and every number on the page has to say which one it is over. Write their definitions into the record before anyone is contacted, then leave them alone.

The two most abused are the third and the fourth. A screener written after the results arrive will always improve them, so write it down first and freeze it.

A conversion rate over "asked" measures the offer; the same rate over "reached" mostly measures the channel. Reporting one as the other is the single most common way an offer test lies.

First sitting: the decision rule

The rule is one sentence, written before the test runs, containing a count and an action.

It reads like this: if fewer than a stated number of qualified people asked commit, we change the offer rather than the price, and we do not run this channel again until we have. Not a percentage, because percentages over small denominators are how a rule gets argued away. Not a feeling, because a feeling cannot come back negative.

Two properties are worth insisting on. The rule names which variable moves on a negative result, so that failure produces a specific next action rather than a discussion. And it is dated and left unedited, so the result stays readable. Designing a test that can actually fail is the argument behind offer testing as a whole.

Second sitting: the counts

Counts go in as a fraction over a stated denominator, never as a bare rate. The record needs one row for each of the outcomes below, so that nothing has to be reconstructed from memory.

Saw and stopped, with the step where they stopped. Asked and refused. Asked and deferred, which is not a refusal and not an acceptance. Committed to something that cost them, whether money, a scheduled block of their time, or a signature. Committed and then withdrew before delivery, which belongs in its own row because it means something different.

The last row is the one people fold into acceptances. A person who paid and then asked for the money back before anything happened has told you the offer reads better than it holds up, and that is a finding you cannot recover once the row is merged.

Second sitting: refusals in their own words

Record refusals in the person's own words first, then code them. Never the other way around, and never from a dropdown offered to the person.

Offering someone a list of reasons does most of the work of answering for them, which is the well documented effect that survey question wording has on the answers a researcher gets back.

The price refusal and the value refusal sound identical in the room and separate with one follow-up question about whether they would pay that for something that solved the problem completely. That separation decides the next move, and getting it wrong sends you into a price cut when you had a value problem. Pricing validation questions give the full method.

Code wrong-fit refusals honestly and separately, because a test full of wrong-fit refusals has not tested the offer at all.

The drift field and the delivery window

The middle field of the record matters more than it sounds. Tests drift while they run, and a test where the screener quietly widened or the price was softened on two calls is a different test from the one described.

Writing the drift down is what stops next month's reading of the page from being wrong. It costs one line per change and it is written in the second sitting, while the changes are still remembered.

Where money changed hands before delivery, note the delivery window you stated as well. A stated window is an obligation, and the FTC's Mail, Internet, or Telephone Order Merchandise Rule sets out what you owe a buyer when you cannot meet it: notice, consent, or a prompt refund.

Reading two records against each other

The point of the bookkeeping is comparison, and comparison requires that only one thing changed.

If the price moved, the screener must not have, and if the channel moved, the offer wording must not have.

When two things moved, you have two tests and one number. The honest entry is that the result is unattributable. Saying so costs a paragraph and saves a quarter. It is the same discipline that keeps minimum viable offers readable when the delivery is being cut at the same time as the promise is being written.

Why the record holds no benchmark

A conversion figure from anywhere else was produced by a different offer, at a different price, through a different channel, to a different audience, at a different time. Applied to your test it will be internally consistent and irrelevant, and it will feel authoritative, which is worse.

Derive your comparison number from your own arithmetic instead. Market validation metrics give the identity behind it, and it yields the rate you need, not the rate somebody else got.

Common questions

How many people does a test need?

Enough that one person changing their mind does not flip the conclusion. Run that check explicitly: take your result, move one event, and see whether you would decide differently. If you would, you have a hypothesis rather than a result.

Can I count someone who said yes but has not paid?

In its own row, called deferred. Folding it into acceptances is how a test with two real commitments gets reported as a test with seven.

What if the channel produced almost nobody?

Then the test measured the channel and not the offer, and the offer question is still open. Say that, and do not let the small number be read as rejection.

Should the price appear in the first message?

For a test of the offer, yes. A response gathered before anyone saw a number tells you about interest, and interest is what the whole exercise exists to move past.

Do I need the same denominators for every test?

The same four, defined the same way, or you cannot compare across tests. Their definitions belong in one place, written once, rather than reinvented per test.