Wednesday, September 07, 2005

How good are your stats?

I am currently looking carefully at the statistics of the Lidcombe study. I have already done some work months ago. The big big problem with evaluating children therapies is that you never quite know whether they recover naturally or due to your intervention. [Another issue is whether the recovery due to your intervention happens faster than the natural recovery.]

For example, let's say the natural recovery rate is 70%. Then roughly 700 kids out of 1000 kids should recover. Let give treatment to 20 kids, and 16 kids recover. This makes a recovery rate of 80%, which is 10% higher than the 70%. But it could well be a statistical fluctuation as you have picked 20 kids randomly for the population of all kids that are dysfluent. So by chance you might end up with 16 kids that will eventually naturally recover. The trick is either large sample size or large treatment effect. Thus, the larger your sample of kids, the less likely the natural recovery rate in the sample deviates from the 70%. And the larger your treatment the clearer you deviate from the 70%. I derived a mathematical formula to compute the chance that a result is a fluke. And the higher the recovery rate of the sample you have treated and the larger your sample size, the lower the likelihood of a fluke. Do you follow me??? :-) You can also do it with a Monte Carlo simulation, but this gets a bit technical!

Here is my rapid response. I found already a small spelling mistake. I hate when that happens. I read it several times and then I sent it off and then I spotted the spelling mistake... I am curious what others have to say about my response. I guess 90% have no clue what I am talking about. So I hope for the 10% to either agree or prove me wrong.

No comments: