Home > Scholarly Communication > The importance of the journal (peer review) system

The importance of the journal (peer review) system

Problem Statement: An economics professor blogs on the Open Access concept and its supposed role in the death of the journal (peer review) system(!).
Motivation: As someone who has been researching the concept of scholarly communication (including the concepts of Open Access and the (journal) peer review system) I feel it is necessary to provide a much more realistic and supported view of the expected (and already measured) advantages of Open Access and the importance and sturdiness of the journal (peer review) system.
Findings: Even those well versed in Economics can be utterly confusing and nonsensical in other fields.

OK, so I cannot remember exactly what I typed in Google to get me to this post by a Mr. Henry Farrell on Crooked Timber

“Lemme guess: incentives for reviewing?”

But it indirectly lead me to this post by a Mr. Tyler Cowen on Marginal Revolution.

“And…?”

And I must say that is one bizarre post he makes, very bizarre. And I feel it is my duty to give a better share of the context than he did while explaining his rather radical statements.

First of all, I feel there is a very significant lack of understanding concerning the Open Access concept. I think people really need to read at least two resources before they should even think of criticizing Open Access (especially when the criticism lacks any kind of arguments). The first is Peter Suber’s “Open Access Overview” and then there is Stevan Harnad’s “Primer on Peer Review, Payment and Publishing” (mirror link)

Now, I could show some good links on the importance of the journal (peer review) system, too. But frankly, I think its importance is so significant/obvious that I do not actually need links to have that point crystal clear, I should hope. Though this blogger’s statements make me think otherwise, so let us dive in.

I don’t envision the free access system as the status quo but free.

I agree, but while I am thinking about more access to (peer reviewed) scientific literature = more qualified people getting more knowledge = better papers and such, what he means is:

Ultimately there wouldn’t be journals……I suspect refereeing might die

Well, that is rather radical. But let us move back a step and see how he explains this statement:

Papers would be ranked directly in terms of status and popularity rather than ranked through the journals they are published in.

“Hey hey (assuming that I cared), I would tell you to go back further man: it’s missing the reasoning behind this statement.”

I am afraid that is it: there is nothing before the first quote and this quote. He immediately jumps to this conclusion. Personally, I have no idea how he came to this conclusion, and he is not making any attempts to explain it, either. In fact, I do not get the idea he gets it himself, because a little later he says:

Ultimately there wouldn’t be journals and this would make a big difference as journals are the current carrier of selective incentives and status rewards.

OK, so he recognizes that journals are the (current) carrier of selective incentives and status rewards. Indeed, there is a strong link between (the existence of) journals and a paper’s status and popularity. Why that would change because the financing of the journal publishers differ is not explained. Somehow this will magically disappear once journals stop charging their readers (for profit)? How does that happen? I do not get this at all, but I would LOVE to hear that reasoning. Because if that can be proven, well that would make a very strong argument against OA, rather than the drivel that is being spread around now by various parties.

It would be easy to refuse to referee, since you wouldn’t fear being shut out of publication of that journal; I suspect refereeing might die.

I do not get this statement, either. Is he implying that there is no fear for being shut out of publication because the journals stop with existing? In that case there is no point in offering to peer review, so that cannot be it. Then is it because they magically stop caring and/or to matter? How does that happen? And as he claims that people will refuse to peer review if they no longer fear being shut out of publication of the journal in question, is he saying he refuses to peer review for journals because he does not see himself submitting a manuscript to that journal? I can see how there are cases like these occasionally, but talking about it like it is the only or most significant reason to accept/refuse a peer review? Seriously?

And if status were attached to the individual paper rather than the journal, who would bother to become an editor?

First of all, status is already attached to the individual paper: it is called the citation count. And that happens to be one of the most significant indicators of an individual paper’s quality. Not incidentally, the citation count is a significant element to determine the status (i.e. quality) of the journals: the journal impact factor. More info on that at ISI – Thomson Scientific.

The JCR (Journal Citation Reports) provides quantitative tools for ranking, evaluating, categorizing, and comparing journals. The impact factor is one of these; it is a measure of the frequency with which the “average article” in a journal has been cited in a particular year or period.

And for his information: one can already search based on citation counts, since a long time ago, too. And believe it or not, but the significance of the journal (peer review) system did not go downhill since then. And the fact that journal subscription prices continue to rise more and more even in the face of technological advances, does not exactly give me the feeling that I am witnessing a dying breed here. In fact, it seems to be growing stronger and bolder instead! And that does not sound economically comforting to me. Thus, I have yet to read an explanation of why Open Access changes any of this. So why would qualified and interested people not bother to become editors? OK, so now we have arrived at his “conclusion” (he has been making them all along, so why stop now, eh?).

In other words, the partial monopolization of for-fee journals makes it possible to produce status returns to motivate both editors and referees.

Yes, by some also known as the status quo. Thanks for the reminder.

Returning to the free setting, refereeing will survive insofar as writing detailed referee comments on other people’s work helps with your own research; it is interesting to ponder in which fields this might hold.

Points for consistency! Now the only things I am missing are the arguments that lead to these repeatedly stated conclusions.

Here is the deal: scholars like rigorous independent (objective) scrutinies of their works, because passing those gives that work instantly a more credible feel (may still not be “perfect”, but it has passed the first scrutiny so others may more than often not have to do it). That is why the journal (peer review) system that has been established roughly 200 years ago still stands: it works, and it is considered the best that we have. There is no net consensus for a replacement of this system: preprint platforms are not considered replacements of journals by most, if not all of the scientific community. However, one consensus on a shortcoming of the journal (peer review) is that it is, to some degree, restricting the scientific community of access to valuable scientific knowledge.

With the growing feasibility of electronic communication, opening up this access is likely a matter of time. That is just the way it is, it will most likely happen, the only question is when. Unrefereed manuscripts (preprints) is a good example of that. It is not triggered by the Open Access movement, it happened because of technological advances and a growing number of researchers/authors despite the existence of the journal peer review system. If anything, its growing number is evidence that the journal peer review system cannot handle the increasing load. And it is not going anywhere even if we pretend it is not there, or even if we denounce the potential of Open Access. It is one big gorilla that is not only here to stay, but here to grow. So the question really is: when it happens, do we want to be caught with our pants down or not? I do not, so let us work towards preparing for the most likely event, instead of resisting it and go drama queen with the pants near the ankles when it does happen.

Surely, one key element to “handle” this load of newly available information is the quality filter. The journal peer review system becomes even more important for the scientific community, as it is widely established that it is the first necessary quality filter. Yet, the journal peer review system by itself is not enough qualitatively (and in terms of speed). Something that was already established with the proposal of the citation count and the journal impact factor roughly 50 years ago by Mr. Eugene Garfield as supplements to the journal (peer review) system. Where people yelling of the death of the journal (peer review) system then? I do not know, but I do know that if it happened, they would have been clearly wrong, given the significance of the citation count and journal impact factors since then.

Indeed, peer review by itself was not enough then, it is not enough now and it will not be enough later with Open Access becoming even more of a reality. Additionally evident by another new and in significance growing quality metric: J.E. Hirsch’s H-Index. By then, additional quality filters will become that much more important. But as indicated, there is nothing wrong with that, it is simply scholarly communicating evolving to the next phase to handle its next biggest challenge.

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: