How Reliable is Scientific Research?

The scientific method was developed as a means to understand objective reality, or more simply stated, to discover the truth about the universe we inhabit. Commonly, it is regarded to consist of four steps:

1.       Observe a phenomenon.

2.       Conceive an explanation for the phenomenon. This is the hypothesis.

3.       Make a prediction about a future event, based on the hypothesis.

4.       Do an experiment to see if your prediction is verified.

If the prediction is borne out by experiment, there is now some confidence that the hypothesis is true.

Of course, it is possible that the experiment gave the results it did by chance, or because it was done incorrectly, or because it was designed it in such a way that it gave the results the experimenter wanted to see. So another step can be added to the scientific method to correct this:

5.       Repeat the experiment multiple times, with multiple researchers, and compare the results of all the experiments.

Each time a new experiment produces results that agree with previous experiments, the level of confidence that the hypothesis is correct becomes greater. When a hypothesis has been confirmed many times by multiple people, scientists call it a theory, and accept it as the best existing explanation of the original phenomenon. The theory is then used to generate new hypotheses, which are tested by more experiments. It is this iterative process that brings us ever closer to the truth.

Peer review is another process that science relies on to ensure that reported research has been done properly. When a paper is submitted to a scientific journal for publication, it is independently reviewed by two or three other scientists who work in the field that the paper addresses. Each reviewer prepares a list of comments and criticisms that are sent to the journal’s editors and to the paper’s authors. If errors or deficiencies are noted, the authors are expected to correct them before the paper is published. Of course, the authors can always withdraw the paper from consideration, and submit it to another journal with different reviewers.

A recent article in The Economist claimed that most published findings are incorrect, and that science as a whole is not as self-correcting as most scientists would like to think. The Economist article refers back to a paper published in 2005 by John Ioannidis, which has become one of the most widely cited scientific papers in history, and provides a persuasive statistical argument for this claim. Moreover, when scientists have tried to reproduce the work of others, failure occurs more often than success, providing empirical conformation of Ioannidis’ claims. The Economist gives several reasons for this, a few of which I will expand upon.

Biomedical studies and clinical trials are large, expensive and complicated affairs. Their complexity alone means that they cannot be reproduced exactly, not to mention that financial backing is difficult to obtain for repetition of work that has already been done. To remedy this lack of replication, the data from such studies is subjected to statistical analysis, which supposedly allows extrapolation of the results to the population at large. This is a mathematically sound assumption if the correct statistical procedures are used, and the experiment is designed with these statistical procedures in mind. However, statistics is a discipline that few scientists fully understand. Often, a statistician is not consulted as the experiment is being designed, and inadequate statistical procedures are applied to the data, giving questionable results.

Inadequate peer review can also be a problem. Reviewers consider the design of the experiment and the results, but rarely do they check the data analysis, because this is often something that would take many months to do. Reviewers often are not expert in statistics either, so it might be assumed that the experiment has been correctly designed for the statistical processes that were applied, consequently focusing the review on the results of those processes instead of their suitability. John Bohannon, a science writer, submitted a totally spurious paper, which contained serious scientific flaws that should have been easily spotted by any competent reviewer, to numerous open access scientific journals. These journals largely are not among the most prestigious, and some charge a fee for publication. In many cases, Bohannon found that a journal simply rubber-stamped the paper with no review. Bohannon also found that 70% of journals that did review the paper accepted it for publication. Reviews that pointed out the paper’s numerous scientific flaws occurred in only 12% (36/304) of those received. Sixteen journals accepted the paper despite poor reviews. To be fair, many scientists do realize that all journals are not equally reputable, and consider the source when determining how much credence to place in particular results. Bohannon’s sting has also been criticized because among other things, he did not include any subscription-based journals, which are generally considered more reputable, in his submissions.

Science writers tend to focus on the results of single studies, selectively reporting results that are perceived to be of popular interest. Most journals are reluctant to accept negative results (i.e., studies that did not verify the hypothesis being tested). Medical journals tend to focus on the more high profile studies, and these are the ones that most doctors read.

It is vital that scientific results be trustworthy if they are to drive national policy, as well as the future allocation of funds to pay for ever more expensive research. At least a portion of American public is perceived as anti-science, and publicizing that most scientific studies are inaccurate only serves to ingrain that attitude more deeply. The good news is that the scientific community has recognized the problem and is implementing corrective measures. For example, Nature, one of the most prestigious scientific journals, has developed a checklist of factors that must be reported in every manuscript submitted to the journal, in an effort to improve reproducibility. Other journals will doubtless follow their lead. If all scientists cannot become expert in statistics, journals must at least insist that the statistical analysis of data be justified as appropriate to the experimental design, employing reviewers who are experts for this purpose. In this era of Big Data, it is becoming even more difficult to do more multiple independent data analyses, but journals must insist that all of the data for a study be publicly available, and the analysis procedures rigorously described. Finally, scientists must change their attitudes, favoring veracity of results over quick and prodigious publication.

The scientific method was conceived as a means for humanity to discover the truth about the universe. It is the still best method we have to accomplish that goal. We must preserve its integrity at all costs.


Photo credit: sneaka / / CC BY-SA

This entry was posted in Science and tagged , , , , , , , . Bookmark the permalink.

One Response to How Reliable is Scientific Research?

  1. tekrighter says:

    Here’s an update on this subject from Francis Collins, the Director of NIH:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s