A Proposal To Improve Peer Review: A Unified Peer-to-Peer Review Platform (part 1)

September 25, 2010 3 comments

Whoa, has it really been a year since my last update? Geez!

“Slacker! Where have you been!?”

I guess I’ve just been distracted. I have, however, not exactly been slacking off…much. As evidence of that I’m presenting our (me and my co-author) working paper that anyone can freely download over at SSRN. (Not giving up on my nick here though)

The title of our working paper is:
Towards Scholarly Communication 2.0: Peer-to-Peer Review & Ranking in Open Access Preprint Repositories.

“Wait, what? If that’s the title of your working paper, then why didn’t you name your blog post like that?”

This blog title is shorter, more befitting of a blog post and essentially conveys the same message: “Peer-to-Peer Review (& Ranking) + Open Access Preprint Repositories” equals our “Unified Peer-to-Peer Review Platform”. And now I’m going to summarize our working paper in such a way that it’ll fit nicely in a couple of blog posts. This way people who don’t feel like reading through 25 pages (right now) can still get the gist of the proposed model. I can’t guarantee that you won’t miss important details, though. In fact, I’m fairly certain you’re going to miss out on good stuff. Let’s start with…

The abstract:

In this paper we present our unified peer-to-peer review model for Open Access preprint repositories. Its objective is to improve the efficiency and effectivity of digital scholarly communication. The key elements of this model are standardized quality assessment instruments, public and private communication channels, special rankings and novel incentives. The model allows scholars to proficiently evaluate both the manuscripts and their peer reviews. These scrutinized manuscripts and peer reviews will then be made available to the relevant parties. These standardized quality assessments allow for new quality metrics for papers and peer reviews. The Reviewer Impact, which represents the peer review proficiency and peer review output of scholars, is one such metric. The model includes diverse rankings for scholars to appear in to receive better odds of having their own manuscripts noticed, read, peer reviewed and cited. Their specific ranking is proportional to their Reviewer Impact and the overall quality of their manuscripts. The Open Access preprint repository model is a suitable foundation for our model because of its high degree of accessibility, but little to no certification of its deposited manuscripts. With this combination we envision a novel, Open Access, peer-to-peer scholarly communication model that functions independently of, but not incompatibly with, the traditional journal publishing model: Scholarly Communication 2.0.

The abstract is easier to understand if you are somewhat familiar with the strengths and the (criticized) weaknesses of the peer review process, (Open Access) journal publishing and (Open Access) self-archiving. Even better if you are also familiar with the common pitfalls of (recent?) initiatives for improving scholarly communication/peer review. Examples of those pitfalls are the lack of (1) accountability, (2) creditability, (3) scalability, (4) a business/revenue model, (5) efficiency, (6) unique/competitive services and (7) incentives. If you aren’t familiar with any of this there’s nothing to worry about, as there’s always the introduction!

“…and the rest of the paper. And this blog. And e-mail, optionally? Alright, let’s do this!”

Scholarly Communication by Journal Publishers
…is the title of the introduction of our working paper. In it, we briefly describe scholarly communication; why journal publishing is still the most established scholarly communication model despite our technological (digital communication, “Web 2.0”) and cultural changes (Open Access/ the “preprint culture”), common criticism of both journal publishing and peer review and our proposal of implementing Open Access preprint repositories with a peer-to-peer review certification function. The rest of the paper is spent on exploring the significance and feasibility of our proposed model. More specifically, we provide our answers to the questions “What is the added value of the model given what we already have in scholarly communication?” and “Is the model technically, functionally and financially possible?”. Let’s start with exploring the significance first.

A case for a Unified Peer Review Platform (for Open Access Preprint Repositories)
…is the title of the next section of our working paper. In this section we present the opportunities of journals and scholars collaborating on peer reviewing manuscripts on a single, digital platform. One such opportunity is a more efficient reuse of peer reviews by making them accessible to other journals and peer reviewers. Another is the insight in the quality and output of peer reviewers, which allows for a more effective allocation of scholars for peer reviewing. Then there’s the ability to introduce, encourage and provide peer reviewers with sound peer review practices and peer review instruments so they can peer review more proficiently. These services together enable more objective assessments and metrics that put the quality and overall output of peer reviewers in perspective. These metrics allow more proportional rewards while preserving the anonymity of peer reviewers. In short, we argue that a global peer review platform has the potential to improve the accountability, manageability, effectiveness and efficiency of the certification function of scholarly communication.

“Sounds nice.”

There’s a big bottleneck, however..

“Which is?”

By and large, journal publishers will not collaborate on this. There are simply little to no incentives for them to do so: sharing peer reviews that their journal editors initiated and managed weakens their respective journals’ competitive position. As a journal’s competitive position is greatly based on their ability to identify good or potentially good papers through peer review. Another disincentive for journals is that a place for scholars to submit their manuscripts and carry out peer reviews which other journals can peek at (eventually) levels the playing field quite a bit. And one significant advantage that high profile/impact journals have is receiving first dibs on (potentially) significant manuscripts for publication. So there are even fewer reasons for these journals to voluntarily cooperate with leveling their playing field. And without the participation of high profile/impact journals, a unified peer review platform is suddenly a whole lot less attractive for scholars, who generally want to see their papers published in high profile/impact journals.

So the case is not that we don’t want to accommodate journals when it comes to improving peer review in the aforementioned ways. The reality is that it’s simply not realistic to expect journals, especially the high profile/impact journals, to agree on collaborating on a unified peer review platform. Of course, if other (well respected) journals want to cooperate, that would still turn it into a beneficial service to provide, but we’d like to think that the issue of weakening a journal’s competitive position is relatively the same for all journals except those with the lowest impact factors.

In addition, what we’re trying to achieve with a unified peer review platform is not just that sound peer reviews can be more efficiently utilized by the relevant parties. As valuable as that would be, we additionally want to increase the insight and management of the peer review process. We want to have a more objective way of evaluating a scholar’s peer review proficiency and output. And for that to happen an additional requirement is that every single peer review must be scrutinized in roughly the same way. Only then can we say that we can more accurately compare the peer review proficiency (and output) of scholars.

There is additionally the matter of journals having different quality standards and methodologies to reach those standards.
Getting all the different journals to agree to apply the same standards and criteria is likely going to be difficult, if not impossible. It is, however, a necessary requirement to ensure peer review consistency and enable objective assessments and comparisons.

“Now you’re just being greedy. If you can manage to have journals and scholars agree to share their peer reviews, isn’t that already a valuable enough outcome? Sounds like a worthy goal to strive for.”

But there’s nothing original about the idea of a unified peer review platform where scholars and journals can share their peer reviews to improve the overall peer review process of manuscripts. The fact that something like that doesn’t exist between journals not from the same publisher/brand (as far as we know?) means there is at least one fundamental issue that’s preventing scholars and/or journals from doing this already. And given the highly competitive industry that is journal publishing, we’re assuming that the aforementioned disincentives of journals to collaborate on such peer review platforms are more right than wrong.

“Oh well, that’s too bad. Nice try, though. Back to hibernation for a year?”

But the potential of a unified peer review platform to improve peer review is, in our opinion, too great to simply leave it at that. We also have more options when it comes to digital collaboration.

From a different perspective, Open Access preprint repositories have the potential to meet the mandatory requirements. They have openly accessible manuscripts, which are eligible for peer review. They are established platforms for scholars to visit and make use of. They have little competitive desire and instead are more open to utilizing open standards and newer communication and collaboration technologies to make research as accessible as possible. They already fulfil three of the four main functions of scholarly communication. The only function repositories are traditionally missing is the certification function. A lack of a certification function results in a lack of the journal editor role, a key role in traditional peer review process. For that reason, Open Access preprint repositories are not eligible for a global peer review system with the way peer reviews are traditionally carried out. However, they are eligible for a global peer review system utilizing a peer-to-peer review system instead.

Services provided: Repositories versus Journals
So that’s how we shifted from “journal peer review” to “peer-to-peer review (in Open Access preprint repositories)”. Significant challenges are still unaddressed here, but that’s what the rest of the paper and the next blog post are for. One last thing I want to point out before finishing up the first part of this blog series and this section of our working paper is this:

However, that does not mean it should be incompatible with the current journals models. There is no practical advantage in denying the traditional enablers of (journal) peer reviews from participating. Particularly when considering their established value to scholarly communication, compatibility increases the utility of the peer-to-peer review model.

We see efficiency an important element of our proposal. Allowing for maximum compatibility with already established scholarly communication models means that the time and effort that scholars contribute to this environment won’t be wasted, regardless of whether a peer-to-peer review model (like ours) can function independently or not.

Concluding part 1
So there you have it. That’s the context of our proposed model. The next section of our working paper is titled: A General Overview of the Model: Simulating the Journal Editor. In this section, we briefly describe the functional environment that is to support our peer-to-peer review process. After that we can finally dig into the meaty stuff: detailing how the actual peer-to-peer review process is carried out, how such activities can be properly credited, how to encourage professional behavior, how to enable accountability, provide “career improving” incentives and how all of this leads to new metrics for authors, papers, peer reviewers, peer reviews and ultimately a more effective and efficient way of scholarly communication. In the Discussion & Conclusion section, we’ll go over all the “opportunities” and “threats” (to use it in SWOT terms) and talk a bit about how this model can gain revenue. Feedback would be most welcome!


WordPress Post Pagination

August 8, 2009 Leave a comment

Thanks to a couple of helpful sites [1, 2]: I have learnt that it is apparently possible to paginate a single blog post. Extremely useful for long blog posts. Of course, the actual tag “Next-Page Quicktag” is not actually found on my “Write Post” interface, so I have to insert the code for “Next Page” manually to get it done. This is not a big deal, but I do wonder why they have not just included it in the interface.

Something I just felt sharing, in case others wanted to know. I have been going through some of my older posts to get some pagination done. Just to make navigation easier. I hope it does not backfire and actually make me lose the few viewers that I had because they rather not click on the “Read the rest of this entry »” links or the links to the respective paginated pages of a single blog. I guess that is why some bloggers just put up with huge posts on the “front page”: easier to scroll than to click?

Categories: Web (2.0)

On Digital Publishing and Readability

August 2, 2009 Leave a comment

While it did start a discussion on a topic that I am interested in reading, I am not going to spend time dissecting “Reinventing academic publishing online. Part I: Rigor, relevance and practice”. However, there are two things I wish to say about this article regarding its content. The first point is that I am not going to bother reading this article (which is part 1 of their paper), but I am very much looking forward to reading part 2 of their paper.


Because there are only so many “these are the problems” articles I am willing to read and comment on before I want to read nothing but “here are our/the solutions to these problems” articles. And the second point is that I think it is very ironic how more and more journals are publishing articles that talk about the limitations of the journal publishing models. And, correspondingly, how there is a need to improve them to fit with the reasonable but growing demand for a higher degree of quality, efficiency and effectivity of scholarly communication.

“Smells like “friendly” fire to me.”

Of course, in their defense: scholars generally still consider the (current) journal publishing models the best they have for scholarly communication. So scholars publishing in journals is (still) justified. And journals operating the way they currently do is also (still) justified. The latter not without growing pressure to do better, I think/hope.

Anyway, this article actually triggered a discussion on the issue of readability on FriendFeed.

“But nothing on the content itself? Awkward…”

I cannot constructively comment on the (lack of) readability of this article since I have not read it. Yet, I did notice the use of bulleted lists as I skimmed through the article, and there is something I wish to share. See, I was once told by one of my professors, as a critique on a report I submitted, that using bulleted lists was generally not recommended. Apparently, bulleted lists take the “flow” out of (reading) the text. That was a sad day for me, since I really like bulleted lists. To me, it makes reading/absorbing the information easier/faster, but I could see my professor’s point, too.

“Now you have to learn how to write better. Bummer.”

Also, they read all their stuff printed/analog, while I try to read as much (and fast) off the computer screen as I can (saves ink, printing can be expensive!). I wonder if that can make a difference, too. Maybe bulleted lists are just better for people who aren’t solely focused on reading something, while they can be an annoyance to people heavily focused on the text.

“It’s probably just the ADD kicking in for Internet Junkies like you.”

Indeed. Maybe I should just try to read more printed stuff, away from my computer.

“That is a natural progression from “IT bum” to “adult”. I guess there is still hope for you.”

Revisiting the value of blog posts and blog comments

June 20, 2009 2 comments

Here is a blog post by Mike Taylor on the issue of crediting qualitative contributions of a scholarly nature titled “Yet more uninformed noodling on the future of scientific publishing and that kind of thing”. The blogosphere seems to have more of these “rating blog posts/comments” discussions lately. Considering I am replying to that blog post, I could have aptly titled this blog post something like “Citing and ranking web based scientific documents”. However, it was actually a different issue addressed in that post that compelled me to write this.
Read more…

Slow Blogging versus Speedy Twittering

April 1, 2009 1 comment

A bit late perhaps, but as I am chatting away with a friend who Twitters, I realize one of the benefits of Twittering over Blogging (or any other type of “authoring” really).

“Let me take an exceptionally educated guess: you not actually being a member of the Twitter community? Sounds great indeed; where do I (and, for that matter, the rest of the world) sign up?”

In your dreams and my nightmares. See, one of the principles of successful change management is to, as quickly as possible, achieve success. Reaching that milestone, no matter how small is it, will give people the feeling of being productive and useful. Reaching milestone after milestone, no matter how small they are, is an experience that strongly motivates people to (continue to) do things.

There are many times when I thought of something quick that I wanted to write about, but refrained from doing so because my mind went “I cannot just write stuff down. I will have to take my time and think about it, then write a draft, then spend quite a bit of time reviewing and rewriting that draft into a coherent and complete story”.

“Huh? When have you ever used that procedure for this blog?”

I am doing it even now. So the idea that I had to spend a lot of time to share what I felt was a very little nugget of wisdom was not exactly motivating. Which is kind of insane since blogging was the Twitter before Twitter! It was an easy outlet of whatever one wanted to say online right there and then, quick and dirty! Format be damned! But when blogging became a more serious activity for professionals, the people behind Twitter had to look for the next format where “quick and dirty” was the rule, no exceptions. By limiting it to 140 characters, they have successfully captured, no… abducted the “quick and dirty” feel and complemented it with the “repeatedly strive for quick achievements, no matter how small” concept.

No wonder they called it Twitter: it sounds and means something fast, whereas “blog” sounds slow and its definition reminds me of something slow. If there ever was an alien called “Blog”, you know it would be something big, fat, wobbling with multiple layers of whatever.

“Hey, leave the snark to me! Anyway, abducted is a strong word; I predict a new service that will have even a lower maximum limit of characters reaching great heights of popularity!”

Not an impossible theory. Still, how articulate can one be with even fewer characters? Wait, do I even want to know?

Wobbling Blogger Out.

Categories: Web (2.0) Tags: , ,

Wobbly Update: Sprouts: Working Papers on Information Systems

February 8, 2009 2 comments

Just a small update; I have been rather busy with stuff so I have not been able to update my blog properly. I have been and still am working on a very big blog post, though. And I guarantee you that that blog post will be very awesome. Until that time, a small update on something interesting that I have found through my FriendFeed contacts. Looks like the number of repositories have increased by one: Sprouts

Sprouts: Working Papers on Information Systems is an Open Access publication that provides a fast-turnaround outlet for authentic and original research and work-in-progress carried out by scholars of the information systems field and members of AIS, the Association for Information Systems. Sprouts is devoted to research about the ways in which information is generated and used in the prevailing complex socio-technical landscape.

Sprouts is special because it focuses on Information Systems. Which is still relatively a new field to embrace the Open Access preprint culture. I know Social Science Research Network has a special section for (working) papers on Information Systems, but having one specifically for Information Systems sends a stronger signal in terms of recognizing the value of OA preprints. At least in my opinion anyway.

Interesting progress for the Open Access theme!

A new category: Poking At New Research

December 21, 2008 Leave a comment

Blog Abstract
To keep up with (random) new research, find new material to blog about and practice my skills to poke at said research, a new category for this blog, aptly named “Poking At New Research”, is created. The first entry in this category is an article that focuses on the impact of intentional pain.
Read more…