Rules
Part of When problem discovery stops being useful and starts stalling
Problem discovery metrics: the two counts that do most of the work
Problem discovery metrics that can fall rather than only rise: surprise rate, cold sample share and workarounds found, and why no threshold is offered.
Problem discovery can produce plenty of activity without stronger evidence. Start with existing spend observed, the strongest signal in the record, and surprise rate, which shows whether conversations still change the written statement. Keep counts tied to denominators: there is no universal target, because segment variation and the cost of the next commitment determine what a useful threshold means.
If your records show effort but not evidence
If the log mainly tracks conversations held, hours spent, or notes taken, add measures that can fall. Activity counts rise with effort regardless of what you learn. Keep conversations held as a denominator, then use counts that describe the quality of collected evidence rather than treating volume as success.
| Count | How to record it | What a low count may mean | What it does not establish |
|---|---|---|---|
| Unprompted problem mentions | Conversations where someone described the problem before you named it | You may be explaining the problem into existence | That the problem is not real |
| Workarounds found | Distinct, effortful things people already do about the problem | You may be hearing complaints rather than costs | That nobody has the problem |
| Existing spend observed | People paying, hiring, or subscribing to reduce the problem | No budget line has appeared yet | That a budget line could not exist |
| Buyer contact | Conversations with someone who could authorize a purchase | You may still be talking only to the sympathetic layer | That the enthusiasm is fake |
| Cold sample share | The fraction reached through a route you could repeat | Your evidence may describe your own network | That the audience is unreachable |
| Surprise rate | Conversations that changed the written statement | You may have reached saturation, or stopped listening | That discovery is finished |
| Trigger recall | People who could name the event that made them look for a solution | You do not yet know when anyone buys | That purchases are random |
| Statement revisions | Dated changes to the problem statement | Nothing may have been allowed to contradict you | That the statement is correct |
Among these measures, existing spend observed carries the most weight. People already paying, hiring, or subscribing to reduce a problem distinguish a market from a category. Record this as a count first, with its denominator visible, before converting it to a percentage.
Fractions keep the sample size in view: twelve out of thirty and two out of five produce the same percentage, but only the first gives a sample worth acting on. Ratios from small samples can be noise wearing a percentage sign, so keep the fraction rather than reporting the rate alone.
- Rule of thumb: Keep effort counts as denominators and use falling counts to judge evidence quality.
If conversations keep changing the written statement
Mark for each conversation whether the written problem statement changed, then count those changes. When new conversations still alter it, learning is active; when several in a row leave it unchanged, the segment may be saturated. This count helps decide whether to continue listening or examine reach.
Early in a segment, the surprise rate is high. As you learn, it should decline; several unchanged conversations in a row suggest saturation among the people you have spoken with. Do not read that as knowledge of the whole market, because the count describes only this group.
A low cold sample share changes how to read a flat surprise rate. It may mean you exhausted your own network rather than learned enough about the segment. Those outcomes can happen at different times, but from inside the discovery process they can look the same.
- Rule of thumb: Repeated unchanged conversations suggest saturation of that segment, not proof that discovery is complete.
If most conversations come through familiar routes
Separate each conversation by how you found the person: a personal route or one that anyone could use.
Warm conversations can teach you about the problem; cold conversations test whether the audience is reachable, a question that becomes central in market validation. Record the sample route beside each conversation for later comparisons.
A high surprise rate alongside a low cold sample share means you are learning, but only about people like you. Cold sample share is therefore more than a recruiting detail: it shows whether your evidence extends beyond your existing network. Keep the route beside each record so later rounds can be compared on the same basis.
- Rule of thumb: Use warm conversations to learn about the problem and cold ones to test reach.
If positive numbers look like progress
If a number rises but cannot change your next decision, treat it as a comfort metric rather than evidence. Familiar measures can look encouraging while revealing little about whether the problem is real, costly, or reachable. Each has a place, but none is a verdict on its own.
- Conversations held:This rises with effort and is unrelated to learning. Keep it only as a denominator.
- Positive responses:This measures how agreeable your questions were. A high count with few workarounds found suggests leading questions.
- Waiting list size:This measures the appeal of a promise. It decays; a name from six months ago is not a person waiting.
- Hours spent:This is reliably higher for people avoiding the uncomfortable part of the work.
- Survey completion rate:This tells you the survey was short, not what the answers mean.
Survey wording and question order shape answers, as Harvard survey questionnaire design material discusses. These measures can feel easiest to report, but many positive responses paired with few workarounds point to questions doing the work, not evidence that the problem is real or costly.
- Rule of thumb: Do not confuse a rising activity or response count with evidence that can change your decision.
If two counts point in different directions
Compare a count with another before deciding what it means. Discovery counts become informative in pairs: one measure can expose a gap that the other hides. Use the combination to choose what to investigate next rather than treating either count as a verdict.
- High positive responses with few workarounds suggest your questions are doing the work, not the problem. Rewrite them around past events, as in customer interviews.
- High surprise rate with low cold sample share means you are learning, but only about people like you.
- Flat surprise rate with no buyer contact means you may have saturated users without meeting the buyer. This is a common way discovery ends too early.
- Existing spend observed with no trigger recall means money is there, but you do not yet know when it moves. That is a timing problem, and it decides your channel.
- Rule of thumb:Read counts in pairs to decide what the evidence says and what to check next.
If someone offers you a target number
If a target is not tied to your segment and next commitment, do not treat it as a decision rule. A figure such as twenty interviews avoids the real condition: whether new conversations have stopped changing the statement. Segment variation and the cost of the next commitment both matter.
In a narrow, uniform segment, the statement may stop changing quickly; in a varied one, it may keep changing. If the segment varies and new conversations add different evidence, split the segment rather than continuing to count as if it were uniform. American Community Survey data is one public resource to consider alongside your own counts.
The same caution applies to a suggested share of people who should report the problem: the result depends on how the sample was drawn, something an outsider usually cannot see. Produce your own numbers and write the denominator and sampling route beside them. Compare later rounds with earlier ones, not with an outside figure.
A threshold depends on both segment variation and the cost of your next commitment; without those details, a universal target cannot tell you when to stop. Keep the sampling route visible when comparing rounds, since a changed route may also change what the counts describe.
- Rule of thumb: Compare your own rounds and sampling routes, not your percentage with someone else's.
If you are planning another round
Before the next round, write down which count you expect to move and what result would change your course. Afterward, compare the result with that expectation and record the comparison whether or not you liked it. Set the threshold while neutral, before the result is known.
Name the observation that would make a claim fail before the check runs. The claim by claim loop uses the same discipline. For a compact record, track cold sample share, workarounds found, and whether the statement changed.
A spreadsheet with one row per conversation is enough, and it stays useful longer than people expect. You do not need a tool; the record matters more than what holds it, as the discovery log in problem discovery shows. Compare your third round with your first, rather than comparing your numbers with someone else's.
- Rule of thumb: Set the decision condition before the round, then record the result even when it disappoints you.
If someone asks you to pay
If you are asking someone for money, stop relying on discovery counts as the main evidence. From that point, behavior matters more than description, and the counts that matter concern acceptance and refusal. Move to offer testing metrics to measure those outcomes.
- Rule of thumb: When money is requested, measure acceptance and refusal rather than problem descriptions.
