Home > Scholarly Communication > Evaluating Peer Reviews: How To Identify Good Peer Reviews?

Evaluating Peer Reviews: How To Identify Good Peer Reviews?

Blog Abstract
While scholars generally accept the added value of the (journal) peer review, the actual value of a peer review is difficult to measure. However, given the importance of the (journal) peer review process, it is important for scholars to understand what a good peer review is, so they can work towards peer reviewing more qualitatively. Indeed, when scholars can identify the important qualities of a peer review, it will undoubtedly help them with improving their peer review proficiencies more efficiently. That will result in an improvement of the overall quality of scientific literature and scholarly communication, since it is the most significant process for qualifying manuscripts for publication, which (traditionally) drastically improves its visibility and readability.

To get a better understanding of the characteristics of a “good” peer review, this blog post covers “best practices” of peer review and quality assessment instruments that are specifically designed to measure the quality of a peer review. In addition, in light of determining the practical usage of these best practices and quality assessment instruments, the roles of the authors, peer reviewers and journal editors as suitable users of such quality assessment instruments are examined.

“You know what, there’s something fiendishly satisfying about being able to criticize the criticizers. I like it.”

Peer Review Under The Magnifying Glass
Whether research papers by scholars are suitable for publication in scientific journals depends largely on how valid, significant and original the research findings are. One way of finding that out is through the act of peer reviewing. Peer review is the standard for the quality assessment of manuscripts to determine whether they are suitable enough to be published. The process essentially boils down to having peers scrutinize the research papers submitted for publication. A peer review session commonly utilizes about two peers, but peer reviewing for grants may use more given their more direct significant financial consequences. Peer review serves as a quality “stamp’: they have it done by a small number of their peers so that the other peers/readers do not necessarily have to verify the soundness and relevancy of the science as in dept themselves. Peer review also serves as a quality filter, its “quality ranking” indicated by the journal impact factor (JIF) of the journal.

The journal impact factor is a statistic that is based on the citation count of its previous publications, which allows for an immediate indication of the quality (of future publications) right after their publications. I am not going too deeply into the JIF element, but let us just say that this approach to determine the quality of the publications is widely considered as convenient, but not very accurate to even possibly misleading in determining the quality of individual publications.

“That sounds like the fun part that we should be discussing!”

Nah, I rather not open that can of worms, and that would be kind of off-topic. Alright, so a (journal) peer review is about scholarly peers scrutinizing and addressing the good and the bad of a manuscript, to support a journal editor with deciding whether the manuscript can be published (after revision). So to understand what a good peer review should cover, we need to understand what a good manuscript should cover. After all, if you cannot tell a good manuscript from a bad manuscript, then you are definitely not suitable to peer review manuscripts proficiently enough to be of any use. According to a discussion paper by Brown et al. [2002], referees usually comment and make recommendations on some of the following:

  • Significance: Are the findings original? Is the paper suitable for the subject focus of this journal? Is it sufficiently significant? (Is it a ‘me too’ paper; is it ‘salami slicing?)
  • Presentation: Is the paper clear, logical and understandable?
  • Scholarship: Does it take into account relevant current and past research on the topic?
  • Evidence: Are the methodology, data and analyses sound? Is the statistical design and analysis appropriate? Are there sufficient data to support the conclusions?
  • Reasoning: Are the logic, arguments, inferences and interpretations sound? Are there counter-arguments or contrary evidence to be taken into account?
  • Theory: Is the theory sufficiently sound, and supported by the evidence? Is it testable? Is it preferable to competing theories?
  • Length: Does the article justify its length?
  • Ethics: In papers describing work on animals or humans, is the work covered by appropriate licensing or ethical approval? (Many biological and medical journals have their own published guidelines for such research.)

So there you have it, these are significant characteristics of a good manuscript that peer reviewers need to address when peer reviewing. A more recent paper that covers these points and more is a paper by Davison et al. [2005].

“Ehm, doesn’t that mean you’re done with this blog post already? Using other people’s material, too! I can’t say you’re scoring too well on the original factor, buddy!”

Not exactly. Now we know what to assess, but we also need to know how to assess peer reviewers that (may or may not) focus on the above (or similar) characteristics of a good manuscript/peer review.


Pages: 1 2

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: