A new year is upon us, and that’s always a great time to clean out the skeletons in your closet. So without further ado, let’s take a look at Jonah Lehrer’s explanation of “the decline effect” (published in The New Yorker last month). Lehrer describes this odd phenomenon whereby statistical significance of previous scientific findings seems to decrease with age, as we get further and further away from the time that it was initially reported in literature.
As any scientist can tell you, the holy grail of an experiment is a low p-value, a statistical measure that tells whether your findings are indicative of an actual effect, not just randomness and chance. This sounds fairly straightforward – of course we want to find things of actual importance, rather than being lulled into a false discovery by arbitrary data – but it turns out to be much hazier than a simple “yes” or “no.” P-values depend on a number of factors that can change the statistical outcome of your experiment. Things like experimental design, subject choice, even the time of day can have drastic effects on the results of an experiment.
Scientists’ answer to such imperfections is to run the experiment over and over in a number of different environments. This is the beauty of scientific empiricism; at its best, it has the ability to extract truth from the noisy world around us. However, as Lehrer notes, there is one variable that we never change: the fact that people are the ones running these studies. This statement may seem annoyingly obvious, but it’s incredibly important to consider for any scientific study. While the empirical process is designed to provide an objective method of analyzing data, humans are inherently imperfect at being objective and unbiased, and this can manifest itself in the conclusions we take from our studies.
Suppose that you run a study with 90 subjects. The first 80 subjects show a fantastic result. You eagerly begin working on your forthcoming journal article, ready to share your findings with the world. However, upon running the final 10 subjects, you find that this result almost totally disappears. Bummer. An objective machine might say “maybe there isn’t anything here after all” and move on. But people aren’t objective, and they’ve got a stake in giving the world something that is deemed significant. So you decide to leave out those last few subjects, citing them as outliers and thus non-representative of the general population, and publish an article. A few years pass, and a number of researchers (less invested in your discovery) decide to take another look at that paper. They replicate your experimental design, but they fail to replicate your stellar results.
I don’t mean to cast a shadow of doom and gloom over the scientific enterprise. I just want to remind everyone that as long as human beings carry out scientific work, human faults will continue to plague our results. If we hope to come to an understanding of the world around us, it is important that we accept and anticipate the flaws inherent in our system.
via The New Yorker