Thursday, October 16, 2008

The tell-signs of flawed research

Back to John Ioannidis's work (see my post) and his tell-signs of flawed research. Let's see how much is true in stuttering research.

Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.

Absolutely, most samples are less than 30. He recommends 1000s!

Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.

Most research does not include the effect size, but only look at statistically significant differences. If they did, they would find small effect sizes.

Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.

Yes, most studies look at many different variables making it more likely that some correlate by chance.

Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.

Stuttering is very difficult to quantify unlike weight for example. It is a moving target, because people who stutter can fluctuate dramatically. Compare this to a weight measurement where the difference in weight between morning and evening is probably just a kg or so.

Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.

There are great financial incentives for stuttering medication: a potential market worth 100s of million dollars. There are financial incentives in AAF (altered auditory feedback) devices, for example SpeakEasy devices. They are pushing very hard on spinning the evidence. Just have a look at their website. Regarding conventional treatments including Lidcombe, there is some money to be made but the sums involved are peanuts in comparison. It is more a matter of justifying their existence as researchers and clinicians (giving their life a meaning and purpose) than about pure financial gains.

There are certainly prejudices. This is especially true for the clinicians who test their own treatment, or you try to validate their long-held beliefs. Compare this to a geneticist who studies the genetics of stuttering (he has no prejudice on what he wants to confirm). He has no hypothesis.

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

The two hot areas I see are: Lidcombe treatment and emotionality/sensitivity studies... This is very different to the hot area of brain imaging or genetics where it is hot to do research but no-one knows what should be found!

No comments: