Home > Scholarly Communication > Revisiting the value of blog posts and blog comments

Revisiting the value of blog posts and blog comments

Here is a blog post by Mike Taylor on the issue of crediting qualitative contributions of a scholarly nature titled “Yet more uninformed noodling on the future of scientific publishing and that kind of thing”. The blogosphere seems to have more of these “rating blog posts/comments” discussions lately. Considering I am replying to that blog post, I could have aptly titled this blog post something like “Citing and ranking web based scientific documents”. However, it was actually a different issue addressed in that post that compelled me to write this.

“Which is?”

Blog posts versus (journal) peer reviews

As things stand at this point, we have a hierarchy of sciency documents. At the top (which we’ll call level 1) come papers. The reputation of papers is largely determined by formal pre-publication reviews (which we will therefore classify as level 2) — and, increasingly, also by blog posts about the paper, which are also level 2.

Here Mike Taylor gives a simple classification of the (perceived) value of scientific documents.

“So?”

Now, if a paper (postprint) is a level 1 and a formal prepublication peer review is a level 2, then a blog post about that paper is unlikely to be a level 2. At least not in many (dare I say most?) cases. One issue I have with it is that a blog post about a paper is usually a post-publication “scrutiny”, after formal prepublication reviews have already been carried out. And there is a significant difference between prepublication scrutiny and post-publication scrutiny in terms of the time, effort and proficiency required of the peer reviewer.

Journals have minimum standards for their publications. One of those standards is that the documented research has to be scientifically sound. Therefore, prepublication reviewers have to verify that everything written in the paper is sound and does not contain (scientific) mistakes. This results in some constructive feedback on how to improve the paper. And all that in a systematic and timely fashion. As for similar standards for the post-publication scrutiny (by blogs): what standards? Most of these blog posts contain a short summary of the paper with a little bit of personal input as to why they find the paper interesting. They cannot be compared to blogs with no quality assessment standards.

Formal prepublication peer reviews have people called editors, whose incomes depend on a job well done, to supervise and encourage peer reviewers to peer review qualitatively. No such quality threshold exists for blog posts/comments. So even if blogs have quality assessment standards defined, they would still have to find qualified people to (objectively) enforce or even verify those standards in the same systematic and timely fashion. For blogs, there is currently neither a quality assurance of the documents nor incentive to scrutinize them properly.

“You’re rather obsessed about this.”

This may seem like some lame nitpicking over a simple classification of containers for scholarly knowledge, meant as a simple example to demonstrate the hierarchy of said containers, but we should be very clear about the differences between the concepts we are talking about. I get antsy every time someone tries to legitimately estimate the value of blog posts up there with papers. And now I have realized that I also get antsy when someone tries to do the same for formal prepublication peer reviews and blog posts.

I think making a case for the value of blogs for scholarly communication is fine. There are, however, some very clear distinctions; blog posts are currently just not as valuable as peer reviews and we should not give off that impression until specific quality standards for blogs are defined and enforced. Until then we should not even accidentally put them on the same level. It will only confuse the scholarly communities even more, and might hurt whatever future attempts people make to promote the value of blogs (and formal prepublication peer reviews). Convincing the “traditional” scholarly communities about the value of blog posts/comments and putting them right up there with papers or formal prepublication peer reviews is disingenuous at best and destructive at worst. It will not make anyone take you seriously.

“Geez alright already! So what else is this guy talking about?”

On citing and ranking web based scientific documents
Mike Taylor then suggests that combining “web based citations”, similar to how paper citations function, and a Google’s PageRank like system might be an answer to the issue of crediting digital containers of scholarly knowledge. But he has identified at least three issues with this approach.

The first is that formal prepublication reviews are not shared on the web, so they cannot be properly cited (through web based citing mechanics). Blog comments, although commonly publicly available, are not on separate pages but usually as an extension of the blog posts in question. That makes citing and processing the citations according to this ranking suggestion more complicated, if not currently impossible.

The second issue is that it has the same problem as paper citations: indifferent to context. Are people citing them because they are good and thus valuable to advance their own points or because they are embarrassingly bad?

The third issue is that, although Google is in probably the best position to make these kinds of citation/ranking reputation systems happen, Google is a private corporation and probably not the best choice to let them handle such reputation management systems.

To address the first issue, we would have to think of something like open peer reviews. But open peer reviewing is actually not that popular. So that move is rather unlikely without some drastic (cultural) changes. For blog posts and comments I can imagine this being a bit easier to solve, however. The second issue is a technical one and one that I cannot imagine it being unsolvable with the right IT infrastructure. The last issue needs a scholarly organization with enough financial and “cultural” support to make these changes happen.

So that is that. While going through some of these discussions, I actually ran into several posts on improving peer review that I thought was also pretty interesting. I think I will blog about that next very soon.

Advertisements
  1. June 22, 2009 at 2:32 PM

    Hi, Wobbler, thanks for your thoughts. I agree that pre-publications reviews and post-publication blogs have little in common, so that categorising them both a “level 2” was probably not helpful. All I meant by “level 2” is that both of them, rather than being primary research, are documents _about_ level-1 papers. I probably should have clarified that level 2 encompasses classes 2A and 2B.

    However, I am not sure about some of your other points: in an ideal world, yes, all pre-publication reviews would be substantial documents that are the result of significant scholarship and represent substantial effort; in practice I have had several reviews of my own papers — both positive and negative — that were little more than box-checking exercises, and which did absolutely nothing to improve the quality of the eventual publication. By contrast, in the last year or so I have read (and written) plenty of blog posts that contain solid chunks of meaty nourishment. Of course that varies hugely on a blog-by-blog basis, but the good stuff is there if you know where to look for it, and this is becoming more true with time.

    The world is changing.

  2. June 22, 2009 at 4:40 PM

    Oh, I don’t deny that a lot of journal peer reviews can be less than useful. And the fact that we lack some kind of a global platform to keep tabs of that is very unfortunate, but that is a story for another day.

    However, even without concrete empirical evidence, I dare assume that a higher ratio and a higher amount of journal peer reviews are more effective, efficient and objective than science blog posts/comments in determining the quality of papers.

    Even if blogs are becoming more and more significant in scrutinizing preprints, they will not reach that same level of significance/scale anytime soon. Not until they change to implement and enforce quality assessment standards, anyway. That is the only way to make them effective and scalable for a larger group of people. However, the “problem” is that once people do that, blogs lose their mojo and they essentially just become what journal peer reviews are now. And there is seemingly no added value in doing that when we already have (digital) journal peer reviews.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: