Revisiting the value of blog posts and blog comments
Here is a blog post by Mike Taylor on the issue of crediting qualitative contributions of a scholarly nature titled “Yet more uninformed noodling on the future of scientific publishing and that kind of thing”. The blogosphere seems to have more of these “rating blog posts/comments” discussions lately. Considering I am replying to that blog post, I could have aptly titled this blog post something like “Citing and ranking web based scientific documents”. However, it was actually a different issue addressed in that post that compelled me to write this.
Blog posts versus (journal) peer reviews
As things stand at this point, we have a hierarchy of sciency documents. At the top (which we’ll call level 1) come papers. The reputation of papers is largely determined by formal pre-publication reviews (which we will therefore classify as level 2) — and, increasingly, also by blog posts about the paper, which are also level 2.
Here Mike Taylor gives a simple classification of the (perceived) value of scientific documents.
Now, if a paper (postprint) is a level 1 and a formal prepublication peer review is a level 2, then a blog post about that paper is unlikely to be a level 2. At least not in many (dare I say most?) cases. One issue I have with it is that a blog post about a paper is usually a post-publication “scrutiny”, after formal prepublication reviews have already been carried out. And there is a significant difference between prepublication scrutiny and post-publication scrutiny in terms of the time, effort and proficiency required of the peer reviewer.
Journals have minimum standards for their publications. One of those standards is that the documented research has to be scientifically sound. Therefore, prepublication reviewers have to verify that everything written in the paper is sound and does not contain (scientific) mistakes. This results in some constructive feedback on how to improve the paper. And all that in a systematic and timely fashion. As for similar standards for the post-publication scrutiny (by blogs): what standards? Most of these blog posts contain a short summary of the paper with a little bit of personal input as to why they find the paper interesting. They cannot be compared to blogs with no quality assessment standards.
Formal prepublication peer reviews have people called editors, whose incomes depend on a job well done, to supervise and encourage peer reviewers to peer review qualitatively. No such quality threshold exists for blog posts/comments. So even if blogs have quality assessment standards defined, they would still have to find qualified people to (objectively) enforce or even verify those standards in the same systematic and timely fashion. For blogs, there is currently neither a quality assurance of the documents nor incentive to scrutinize them properly.
“You’re rather obsessed about this.”
This may seem like some lame nitpicking over a simple classification of containers for scholarly knowledge, meant as a simple example to demonstrate the hierarchy of said containers, but we should be very clear about the differences between the concepts we are talking about. I get antsy every time someone tries to legitimately estimate the value of blog posts up there with papers. And now I have realized that I also get antsy when someone tries to do the same for formal prepublication peer reviews and blog posts.
I think making a case for the value of blogs for scholarly communication is fine. There are, however, some very clear distinctions; blog posts are currently just not as valuable as peer reviews and we should not give off that impression until specific quality standards for blogs are defined and enforced. Until then we should not even accidentally put them on the same level. It will only confuse the scholarly communities even more, and might hurt whatever future attempts people make to promote the value of blogs (and formal prepublication peer reviews). Convincing the “traditional” scholarly communities about the value of blog posts/comments and putting them right up there with papers or formal prepublication peer reviews is disingenuous at best and destructive at worst. It will not make anyone take you seriously.
“Geez alright already! So what else is this guy talking about?”
On citing and ranking web based scientific documents
Mike Taylor then suggests that combining “web based citations”, similar to how paper citations function, and a Google’s PageRank like system might be an answer to the issue of crediting digital containers of scholarly knowledge. But he has identified at least three issues with this approach.
The first is that formal prepublication reviews are not shared on the web, so they cannot be properly cited (through web based citing mechanics). Blog comments, although commonly publicly available, are not on separate pages but usually as an extension of the blog posts in question. That makes citing and processing the citations according to this ranking suggestion more complicated, if not currently impossible.
The second issue is that it has the same problem as paper citations: indifferent to context. Are people citing them because they are good and thus valuable to advance their own points or because they are embarrassingly bad?
The third issue is that, although Google is in probably the best position to make these kinds of citation/ranking reputation systems happen, Google is a private corporation and probably not the best choice to let them handle such reputation management systems.
To address the first issue, we would have to think of something like open peer reviews. But open peer reviewing is actually not that popular. So that move is rather unlikely without some drastic (cultural) changes. For blog posts and comments I can imagine this being a bit easier to solve, however. The second issue is a technical one and one that I cannot imagine it being unsolvable with the right IT infrastructure. The last issue needs a scholarly organization with enough financial and “cultural” support to make these changes happen.
So that is that. While going through some of these discussions, I actually ran into several posts on improving peer review that I thought was also pretty interesting. I think I will blog about that next very soon.