Peer Review Week 2016: What is peer review?

By Dean Williams on September 16, 2016

Image: iStock/Meriel Jane Waissman

During Peer Review Week 2016, IOP Publishing is celebrating the vital contribution that peer reviewers play in science and scientific publishing. Throughout the week we will be making several important announcements regarding how we reward the work of our reviewers. JPhys+ and Physics World will also be dedicating our content to the theme of the week: Recognizing Review.

First, the basics. What is peer review?

Peer review is defined by the Oxford Dictionary of English as the “Evaluation of scientific, academic, or professional work by others working in the same field”. Its origins go back to at least the 18^th Century and are usually attributed to the publication of the first scientific journal, Philosophical Transactions of the Royal Society.

But the process then was not as we know it today. In fact, according to a recent Times Higher Education feature, the first Editor of Philosophical Transactions, Henry Oldenburg, commissioned most of the journal’s content through direct correspondence with the most eminent scientists of the day, and made all of the decisions himself on what was destined for print.

Gradually, peer review evolved into a more standardised and truly peer-led process with the introduction of an “Editorial Board” of experts in a given field, consulted on the validity and quality of a scientist’s work. Nowadays, the level of involvement of an Editorial Board varies from complete control of the peer review process and articles published to almost none at all, where the process is managed by full time staff and the Board are only consulted for advice on new initiatives and particularly unusual cases.

The sciences used to be the preserve of those rich enough to be able to afford such an expensive – and often fruitless – pursuit. As the field grew, so did the number of research articles, meaning these relatively small Editorial Boards were simply not big enough to cope with the volume of new research reported. Modern journals typically consult external referees as well as internal Board Members in order to make publication decisions.

According to a recent STM report, in 2014 there were an estimated 28,100 English-language journals, publishing around 2.5 million articles every year. This huge number is expected to continue rising at around 3% per year, meaning the demand for reviewers’ time is at an all-time high.

Despite the prevalence of peer review, it is widely recognised as an imperfect system, prone to biases (both conscious and unconscious) and can often be time-consuming and may even delay important new findings unnecessarily.

Ultimately, the process is designed to improve the written and scientific quality of published research articles, and ensure that only credible and reproducible results are reported. But there is no shortage of high-profile cases where peer review has not done its job, and the expectations and rigour of the process applied by the tens of thousands of journals varies hugely.

Another important consideration is that the vast majority of refereeing activity is unpaid. The theme of this year’s Peer Review Week, recognising review, reflects a growing concern that this vital part of the scientific process is often taken for granted. What can the scientific community, especially Publishers, do to better recognise reviewers? And are there alternative models to consider if the current process is unsustainable?

Several methods exist for carrying out peer-review. These broadly divide into two types: pre- and post-publication. Currently almost all peer-review is carried out pre-publication, that is, an article must pass through review by experts in the field before it is published either online or in print. Post-publication peer review is a relatively new phenomenon and sees all articles published online after basic editorial checks, but before any review by specialists. Reviewers then provide comments on the published article.

Pre-publication peer review

The theory behind pre-publication peer-review is that collecting expert opinions before publication gives the authors an opportunity to improve their article, and gives journals the opportunity to identify and reject low-quality work before it is published. In theory this improves the quality of published work and thus enhances the scientific record. Pre-publication peer review exists in three forms: single-blind, double-blind and open. Currently single- and double-blind are the most common forms. In all three methods, expert referees are invited by an editorial team, and the referee reports are considered before publication. The article is either then rejected, accepted or returned to the authors for revisions. This process can be repeated until the authors, referees and journal editors are satisfied that an article is suitable for publication.

The most prevalent form of review in scientific and much of academic publishing is single-blind peer-review. In the single-blind process the identity of the referees is concealed from the authors. However, the identity of the authors is made known to the referees. The anonymity of the referees allows them to provide frank feedback and, if necessary, reject articles without any concern for repercussions. Allowing the referees to see the identity of authors can also aid their decision making process and help referees identify incremental work. Conversely, prior knowledge of the authors may introduce bias in the referees’ judgement of a paper. It also raises the risk of referees attempting to block or delay work by competitors while remaining safely anonymous.

One alternative to the single-blind process is the double-blind method. In this system neither the authors nor the referees are aware of each other’s identity. In principle this removes any conscious or unconscious bias from the reviewers work – they should only be judging an article on its content. However, this is not always the case, as an experienced referee or one working in a smaller community may well be able to identify authors from their style or simply the field they are working in. This mechanism also removes the opportunity for referees to compare an article against the authors’ previous work, making it harder to identify incremental work.

The third method, open review, does away with the secrecy of alternatives – the identity of the authors and referees is available to both during review and is published with the article. Some publishers go so far as to publish the referee reports alongside the article. Open review is generally thought to encourage reviewers to be more conscientious in work and to provide thoughtful, thorough reports because they are fully accountable for their comments. The flip-side of this is that referees may find it harder to give negative feedback or reject articles when their identity is known to the authors. There is also the risk that reviewers might be lenient to senior figures within fields as a sign of respect.

Post-publication peer review

Post-publication review is a relatively new route to review articles and is only operated by a small number of journals. Unlike pre-publication review, all articles are published online after basic editorial checks. The published article is then reviewed, either by invited referees or by volunteers. Some journals also allow any registered user to provide comments independent of any formal peer review. If an article passes peer review, it is clearly marked as such and is then indexed in scholarly databases.

The advantage of post-publication review is that it does not introduce any delay to publication; articles are available to researchers very quickly after they are submitted. Allowing many people to provide feedback also increases the pool of knowledge and has the potential to provide far more thorough commentary than traditional review methods. The major disadvantage is that all work is published online, regardless of scientific validity or quality, and once an article is available online it can be difficult to retract. This carries the risk of lowering the quality of published research by allowing poor quality or even fabricated work to be published alongside research, thereby damaging the reputation of the scientific community. Some argue that the scientific process has always been, in effect, post-publication review carried out over years through the reproduction and testing of results.

So there we have it, our first post of the week has covered the various options for conducting peer review – all with their pros and cons, none quite ideal. Indeed, many would say that our current process of peer reviewing articles is the worst system, except for all the others.

Keep an eye out for our other Peer Review Week stories on our dedicated web page and at Physics World. We’ll also be publishing more on JPhys+: A live daily counter of how many referee reports the JPhys series has received during the week, “a day in the life of a paper” exploring the stages a manuscript travels through on its journey to publication, and finally an interview with a Journal of Physics: Condensed Matter Board Member John Inglesfield discussing peer review from the point of view of the most vital people – the referees.

For more information on peer review at IOP Publishing, see our referee guidelines and introduction to refereeing.

This work is licensed under a Creative Commons Attribution 3.0 Unported License