Market validation case study: findings and lessons

A market case study usually arrives as a story about a company that grew, with a number at the start that turned out to be right. The number was chosen after the fact from among the many that were available at the time, and the ones that were wrong do not appear in the article.

No companies appear here either. What follows is a single estimate, built twice from the same idea, to show where the disagreement between two honest people actually sits and what to do about it.

What to take away

• Two honest estimates of the same market can differ by an order of magnitude without either being dishonest. The difference lives in one or two links of the chain.
• Build from your own reachable channel upward. A figure derived from an industry total tells you almost nothing you can act on.
• The output is not a number. It is the name of the assumption the number rests on, and a plan to test it.

The situation

An idea aimed at a specific kind of organization. You know who has the problem, you have spoken to some of them, and now somebody wants a market size. This is the point at which most people produce a large number quickly, because a large number ends the conversation.

Build one: from the top

Start with a published figure for the industry, ideally one with a stated method and a date on it, such as the federal series on business births, deaths and survival. Multiply by the share of that industry resembling your target. Multiply by the share who might plausibly buy something like this. Multiply by what you would charge.

Four steps, twenty minutes. Notice what happened. Every multiplier after the first was chosen by you, from a range in which any value would have looked reasonable, and the result inherits all of that choosing while appearing to come from the published figure.

Top-down has exactly one honest use: a ceiling. If the number comes out small, stop. If it comes out enormous, which it always does, you have learned nothing.

Build two: from one customer

Now build the same estimate the other way, as a chain of terms you could each go and check.

Let N be the number of organizations you can identify by name or through a repeatable route you have actually tried. Let q be the share of those who match your screener when you contact them. Let c be the share of qualified ones who buy when they see a price. Let p be what one pays in a year. The estimate is N times q times c times p.

Four terms again, but a different kind of thing. Each is measurable by doing something this month, and each can be discovered to be wrong.

Where the uncertainty actually sits

Write the four terms in a column and mark next to each how confident you are and how you would find out.

• N is usually the best known, because you can count a list by hand. Where some of those organizations are listed companies, their own filings describe the same market in their own terms and are searchable through the SEC's full text index.
• q is knowable within a week of contacting people.
• p is a decision as much as a measurement, and it is testable with a real price and a real refusal.
• c is where nearly all the uncertainty lives, and it is the term people fill in with whatever makes the total look acceptable.

That is the finding. Not the total: the fact that the total is a statement about c wearing three other terms as camouflage. A factor of five in c moves the answer by a factor of five, and a factor of five in c is entirely ordinary between an early offer and a refined one.

What to do about it

Two things, and neither of them is refining the estimate.

Bound the term instead of guessing it. Run the smallest real test that produces a c at all: a specific offer, at a specific price, to a specific batch of qualified people, counting refusals as carefully as acceptances. The result will be noisy, and a noisy measurement is a different object from an assumption. Write the fraction rather than the percentage, so the size of the sample stays visible. The counting discipline matters more here than anywhere.

Present the estimate as a range with the binding term named. One sentence: this is the total if c sits at the low end, this is the total if it sits at the high end, and this is what we are doing this month to find out which. That sentence is more persuasive than a single number to anyone who has done this before, and it is honest, which the single number is not.

Where the two builds disagree

Run both and the top-down figure will be much larger. The instinct is to average them, or to explain the gap away. Do neither.

The gap is information. It is the distance between the people who exist and the people you can reach and convert through a channel you have tested, and that distance is the actual subject of a market check. An idea can be completely right about the problem and have no economically viable path to the people who have it, and from the inside that failure looks identical to being wrong about the problem.

So the useful reading of the gap is a question: what would have to be true for the reachable number to grow toward the total? The answer is usually a channel you do not currently have, and naming it turns a vague optimism into a piece of work.

What a real case study would need

If you want to learn from somebody else's market, ask what would have to be in the account for it to be evidence: the estimate they made before the outcome, the assumptions they wrote down, the ones that turned out wrong, and the ventures that made the same estimate and failed. Almost no published account contains any of that, which is why the genre reads well and teaches little. The same problem afflicts readiness stories, and the reasoning behind one decision is worth more than ten outcomes.

Common questions

Is a market estimate worth doing at all?

Yes, but for the assumption it exposes rather than the total. The exercise pays for itself the moment it tells you which single term the plan depends on, and that usually takes an afternoon.

What if I cannot estimate N?

Then reach is the open question, and it outranks everything else. Building a list of qualifying names by hand is the work, and it is worth doing before anything gets built. Where to look is covered in the sources and their limits.

Should the estimate go in a plan I show people?

Show the chain, not just the product of it. Anybody experienced will ask which term is doing the work, and having named it yourself is the difference between a document that survives the question and one that does not. What else belongs in that memo is set out in market validation.

When should it be rebuilt?

When any term changes by enough to matter, and specifically after the first real test of c. That is normally the first moment the whole picture moves, and it usually moves down before it moves up, which is a much better time to find out than after a launch: see launch planning.