Tuesday, April 19, 2011

Fraud and weak research

Maybe you heard of the scandal where a study "established" a link between vaccination and autism, and parents (mostly mothers) in the UK went nuts and stopped vaccination. This study has now been severely discredited.

Ora made me aware of an interesting article about the quality of scientific research in the psychological and social research area. See Lehner's article.
An interesting article in the New Yorker magazine about a phenomenon in scientific studies. 
Before the effectiveness of a drug can be confirmed, it must be tested and tested again. Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.
But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.
I suggest you read the first page (screen) and then stop and think hard what might account for this effect, before you read the rest of the article.


Anonymous said...

A large chunk of this blog article is verbatim copy-and-paste from the cite New Yorker article. The author does so without proper indication of quotation: no italics, no quotation marks.

Anonymous said...

I second that a large portion of copied and pasted without indication. I thought it was your writing before reading the article.

A possibility is that the drugs have obvious side effects that provide clues to the doctors. So they see a reduction of symptoms that isn't there.

John S. said...

I recommend a reading of this paper on spurious correlations:'


This would be relevant in brain scan studies.

Tal said...

Here is my guess after reading the first page:

When some new scientific finding has not yet been discovered, only the first "positive" results are published because they are notable. As the new finding becomes more entrenched, the probability of results being published showing that it isn't a real effect (or the effect is weaker than previously thought) increase, because they become "bigger news."