9 problem discovery mistakes that can derail your plans

Discovery goes wrong quietly. There is no crash, no rejection, no moment where something obviously fails. You have conversations, they go well, and at the end you know exactly what you thought at the start with more confidence attached.

The nine below are grouped by where they happen: who you asked, how you listened, and what you wrote down. The grouping matters, because the repair is different in each family and because two of the three are invisible from inside the conversation.

What to take away

• Most bad discovery is not bad questioning. It is a sample that could only ever produce agreement.
• The tell for nearly all of these is confidence rising while the written problem statement stays still.
• Fixing the record is the cheapest of the three families, and it is what makes the other two detectable at all.

Errors of sample

Asking the reachable rather than the affected. Your network answers quickly and kindly. It is also biased in a known direction and cannot be repeated at scale, so a yes tells you nothing about whether the audience exists.

How you catch it: count what fraction of your conversations came through someone you already knew. If it is most of them, you have learned about the problem and nothing about reach. Building the sample from a described group rather than from an address book is the starting point the SBA's guide to market research and competitive analysis assumes.

Stopping at the sympathetic layer. The person who feels the problem talks to you willingly. The person who releases money is harder to reach and less pleasant to ask for.

How you catch it: ask each person who signed off the last comparable purchase. If nobody can name that person, you have not met the buyer.

Treating one segment as the market. People in different situations experience the same problem differently, and averaging them produces a description that fits nobody.

How you catch it: when accounts vary wildly, resist smoothing. Split them, write a separate statement for each, then notice which one you actually have access to.

Errors of listening

Introducing the solution too early. Once someone knows what you are hoping for, everything after is a reaction to your idea rather than a description of their life. It is usually done out of courtesy, to explain why you are taking their time.

How you catch it: mark in your notes the point where you first mentioned the product. Anything after that mark is a different kind of data and should not be mixed with what came before.

Accepting a summary instead of an episode. People generalize by default, because generalizing is polite and quick. "We usually handle it in the system" is not an account of anything.

How you catch it: look for a date, a name, or a step in your notes. A conversation that produced no specifics produced a summary, and the details are still unknown.

Hearing the complaint and missing the workaround. Complaints are cheap. The spreadsheet somebody rebuilds every month, the message sent twice, the step done outside the official process: those cost real effort, which is why they are trustworthy.

How you catch it: count the workarounds you found. A round that surfaced none either had the wrong people in it or never got past the tidy answer. Customer interviews covers how to get past it.

Errors of record

Writing conclusions where observations belong. Notes that say "clearly frustrated with the current process" have already done the interpreting, and the raw material is gone.

How you catch it: try to reconstruct what was actually said from your notes a month later. If you can only find your own conclusions, the evidence was thrown away at the time.

Never updating the written statement. It gets written once and then everything is filed under it, including the things that contradict it.

How you catch it: look at the version history. No changes after twenty conversations means either you were right first time, which is rare, or nothing was allowed to contradict you.

Reasoning from the most recent conversation. The last one is vivid and the enthusiastic one is memorable, so decisions get made from two data points out of thirty.

How you catch it: review the whole log before any decision, and deliberately reread the ones you found boring. Boring conversations are where the ordinary case lives, and the ordinary case is your market.

What they share

Each of these removes a way of being wrong. Not deliberately: every one has a sensible reason attached, and the reason is usually true elsewhere. Talking to people you can reach is efficient. Explaining your idea is polite. Summarizing notes is tidy.

The combined effect is a process that cannot produce bad news, and a process that cannot produce bad news is not producing information either. The same shape appears one level up, in the readiness mistakes.

The cheapest defense

Two columns and one sentence.

Two columns in your notes: what the person said and did, and what you concluded. Keep them apart permanently, so that a later change of mind can be tested against the original material.

One sentence before each round: what would disappoint me. Written down, dated, kept. If a round cannot produce that sentence being true, you are collecting encouragement, and the twelve checks will fail at item four.

Common questions

How many conversations before these start showing?

They are present from the first one. The sample errors are detectable immediately, because you know who you called. The listening errors show up when you read your own notes back, which is why writing them up the same day matters.

Is it wrong to talk to friends at all?

No. They are excellent for sharpening questions and terrible for deciding anything. Use them early, keep them out of the count, and never let a friendly conversation be the reason you moved forward.

What if I have already made all nine?

The material is not lost. Reread it with the sample in mind, mark which conclusions rest on people from your own network, and treat those as open questions rather than findings. Then go and get the missing rows: the buyers, the boring cases, and the segments you smoothed over. Problem discovery sets out what a complete record contains.

Do these apply to survey research?

The sample errors apply more strongly, because a survey hides who did not respond. The listening errors mostly do not, since there is no conversation to lead, though the wording does the leading instead, which the guidance on writing survey questions works through in detail. Surveys have a different failure: they can only ask about possibilities you already thought of, which makes them a counting instrument rather than a discovery one. Where they fit is market validation.