Is science in trouble? An insight into the reproducibility crisis

Is science in trouble? An insight into the reproducibility crisis

Yan-Yi Lee

The world of science has never been void of strong claims. “Calcium protects against preeclampsia in pregnant women.” “Ego-depletion is real.” “Bilinguals do better in executive tasks than monolinguals.” As convincing as these claims may sound, attempts to replicate these findings do fail more often than expected — a phenomenon that marks one of the greatest challenges of metascience today.

It is not difficult to understand why things have ended up this way. As Oxford professor Dorothy Bishop eloquently argues, the reproducibility crisis is mostly created by what she describes as “the four horsemen of irreproducibility”: (i) publication bias, where studies that yield “no effect” are deemed less favorable and less likely to get published; (ii) low statistical power, where the peril of small sample sizes cause existing effects to go undetected; (iii) P-value hacking, where researchers misuse data to report only parts that are statistically significant; and (iv) “HARKing”, where researchers form hypotheses only after results are known. Among the four, publication bias is a particularly dangerous practice; it may misguide high-stakes decisions such as policy enactments and even the designs of medical treatments.

Journal-publishing may be a modern-era practice, but reflections on humans’ limited abilities to capture reality could be as old as scientific practice itself (see Greek philosopher Plato’s Allegory of the Cave). Even with today’s advanced methodologies and technologies, it is still inevitable for scientific misconceptions to develop sometimes in the collective pursuit of knowledge. As such, it is instrumental for the scientific circle to acknowledge current unhealthy scientific practices and approach science with a more unbiased lens — an investment that would likely save us gigabucks on flawed research down the line.