The Most-Cited Finding Was Wrong

The most-cited effect across 90+ experiments was wrong.

Not the direction. The direction was right. Specific instructions produce more specific output than vague instructions. That replicated across evaluators, across models, across tasks. The direction survived everything.

The magnitude was wrong. The number was 2.34. Cited in seven pieces of writing. Referenced in thirty framework documents. Built into the theory of how specificity constrains AI output. The foundation number for the most-cited claim across all the work.

It was actually three effects stacked on top of each other, reported as one.

The original experiment compared a 22-word specific instruction against a 2-word vague instruction. "Do not produce a recommendation that could apply to any B2B SaaS company. Every point must be grounded in Northvane's specific situation" versus "Be specific." Twenty runs. Large effect. Replicated.

The obvious interpretation: specificity is the mechanism. But the comparison mixed specificity content with instruction length. 22 words versus 2. Any effect could be the extra instruction text, not the specificity within it.

A length-controlled replication added two conditions: a short specific instruction and a long vague instruction packed with quality demands. The long vague instruction beat the short specific one on total markers. First interpretation: length is the mechanism. The specificity claim collapses.

But that interpretation was also wrong. The long vague instruction produced 48 percent more words. More words produce more specificity markers mechanically. Per word, the specific instruction was more specific. Length inflated the count. Specificity was real underneath. And the long vague instruction contained quality demands that could independently drive the effect.

Three confounded variables across the two replications. A clean decomposition required separating all of them.

The clean test held specificity and quality demands separate, and matched instruction length.

Specificity is the mechanism. Quality demands do almost nothing alone. They lift the score sharply on top of specificity. Together the effect is larger than the sum. The magnitude is 1.34, not the 2.34 originally claimed.

The tool that caught the confound came from the measurement infrastructure I'd already built. Density analysis was developed for fabrication measurement. Applied to the specificity experiment, it caught the confound. The system caught its own error using its own tools.

The measurement tool built for fabrication detection revealed a confound in specificity measurement. Density analysis isn't novel methodology. It's basic normalization that any researcher would apply. What made it possible here was having the measurement infrastructure already built and the habit of applying it reflexively. The confound led to a cleaner experiment. The cleaner experiment confirmed the mechanism at a smaller, more honest magnitude. The overclaim was replaced by a better-supported claim.

Then a domain expert evaluated the outputs blind. Couldn't tell which were produced with specific instructions and which with generic ones. Picked specific 3 out of 5 times, chance level. Both conditions produce the same analytical conclusions. The specificity instruction changes what the output LOOKS LIKE: more data references, more grounded language. Not what it SAYS.

The direction held. The magnitude shrank. And the mechanism turned out to be about verifiability, not quality. Specific outputs can be checked. Generic ones can't. The analysis is the same either way. Three corrections deep. Each one more interesting than the finding it replaced.