Digital Scholarly Communication & Bottlenecks
Yay, I have found a bunch of interesting articles while screening Nature’s blogs. Or to be more specific, this post by Noah Gray highlights some very interesting issues that I have been working on as well. Namely the issue that academic environments specifically made to receive commentary from scholars do not actually receive enough (or any at all) of them. Anna Kushnir of The Journal of Visualized Experiments made a blog post here concerning this lack of love for scientific blogs/articles on the world wide web. She addresses some good points which, by the way, also have been discussed before in this corner of Nature’s blogs: Peer-to-Peer. In addition, this article by M. Mitchell Waldrop at Scientific American generated quite a bit of commentary on this topic as well. Anyway, rather than being just a cheap advertisement for other digital places, here are my 2 cents…
From what I have intensively(!) read the last year on peer review and scholarly communication…
“This is a good time to put up a disclaimer: this blogger has neither published nor peer reviewed a journal paper before. In this situation, he is an “intensive” dreamer and not a doer!”
..lack of time and lack of “real” incentives are indeed the biggest bottlenecks here. Apart from that, there is just a lack of structure and therefore a lack of efficiency to do this the blog way. Blogs are all over the place and there are no standards and no minimum quality screeners. If a journal asks you to peer review, it is normally a paper that has at least been screened by an experienced editor with a “OK this sounds at least worth a review” rubber stamp on it. That helps, but blogs do not come with that guarantee.
Besides, who knows what they do with those comments? OK, they are blogs written by scholars, but how serious are those pieces? Have they been thought out well, or is it just an “OK this is on my mind right now, and I just want to generate some discussion” kind of articles?
Are they truly worth your time then?
Even if their intention is to generate some significant thoughts and make a real paper out of it, how long will that take, and what will be the success rate of it? There are also only few places that have scientific blogs in one place rather than scattered around over obscure URLs that few people will find/visit regularly. Is it still worth your commitment then to search them out and contribute to them? There are way too many “what if” factors associated with blogs that I am pretty sure the whole idea of serious commentary for scientific blogs will not take off anytime soon.
This does not necessarily apply to Open Access organizations such as PLoS and (Open Access) (pr)eprint repositories such as arXiv and Nature Precedings, which have minimum screening and even published articles available. Add to that the interoperability of the various (pr)eprint platforms by way of the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) and it is possible to have search engines such as OAIster as a centralized contact point for these resources. Do not underestimate these protocols and the concept of “harvesters” (read up on this concept here by Swan et al., 2005), as they are dynamic enough to do more than simply aid in interoperability between digital (pr)eprint platforms. (Rodriguez et al., 2006) designed a peer review system that can use OAI-PMH to find suitable peer reviewers and therefore enable a peer review session. Complete with “tagging” these articles that have been peer reviewed.
Thus, it is much more likely that the digital open peer commentary movement will be kicked off by way of (pr)eprint platforms rather than scientific blogs, because it is technically already possible and also scientifically preferable. And with that I mean that the articles are already written in the scientific preferable format> i.e. journal paper style. If you ask me, unless blogs replace journal papers as the default format for scholarly communication, there is no real advantage to having qualified scholars focus on scientific blog posts as opposed to preprints. Course, proof is stronger than argument, so the people at Research Blogging are focusing on making this scientific blog concept more efficient and user friendly to view and filter these blogs on quality.
Then there is the issue of “why spend my time on peer reviewing when nobody is specifically asking me for it, while I am busy with my own research and peer reviews that journal editors(!) have personally requested (and will remember if I say no)”? Peer reviewing is far from a selfish act, but it is not a completely selfless act, either. For one, peer reviewing puts you on the good side of journal editors. And if they believe you are good, they will assign better papers to you, or you can be more lenient with your peer reviewing tasks because you can show them what you have already done and/or are still doing. Posting anonymously on preprints will not do this for you and I doubt “I have already peer reviewed way too many preprints in repository x and y” is a good excuse to them (even if it is true).
Following with that line of thinking, I do not see the big advantage in the way PLoS works: they are inviting commentary on papers that already have been peer reviewed and published (by their journals). Blogs are largely guilty of the same: they are not the standard in communicating new scientific research, that is the task of scientific papers. Therefore, I consider scientific blogs generally more neatly visualized comments on (published) scientific papers. Asking people to review those scientific blogs is akin to asking people to review already peer reviewed and published papers. If we consider the whole concept of “lack of time”, does the scientific community really want potential peer reviewers focusing on (blogs based on) peer reviewed and published journal papers? Is it not more efficient for them to focus on unrefereed/unpublished scientific literature instead? Those published papers will get their “commentary” in the form of other papers carrying on their research based on these published papers if they are significant enough. That is the point of (search engines that filter papers based on) citation counts and journal impact factors. Is that not enough? I just do not see how this will make scholarly communication more productive, having qualified scholars doing more of the same on already scrutinized works. (In PLoS’ defence, I have read an interview by Chris Surridge, the UK-based managing editor of PLoS ONE, that this is their way of replacing the journal impact factor as a quality indicator for (individual) journal papers. Considering that the journal impact factor is generally not considered an accurate quality indicator for (individual) journal papers, I can see sense in their objective. I still think using potential peer reviewers for this is highly questionable, though). Then again, I suppose there are some clear-cut cases where discussions (through blogs) are productive: when it is directly related to the (current) research of the posters/reviewers.
OK, putting that aside for later and focusing on the lack of commitment thing. I think to kick start something like this, there is a need for a widely (and with that I especially mean journals and universities/scholars) recognized unified digital environment (involving preprints, such as Nature Precedings, arXiv etc.) where qualified people can “peer review” and be recognized for what they are doing, while still staying anonymous at the same time.
Depending on how you go at it from a technological perspective, they are not mutually exclusive. I do not want to go into the details of that (yet).
“Regardless, isn’t that just doing it the old way, but then over the Internet?”
Not exactly, one obvious improvement is that you have taken off a “leash” of potential reviewers: rather than peer reviewing when asked, they can peer review whenever there are suitable preprints for them to peer review (on top of being asked, if they feel there is room for that). Also, once that is possible, we have a kind of environment that is more friendly towards more interesting incentives to experiment with to further encourage those qualified to participate. After all, there is no point in doing something for “the open community” when “the open community” is not there to begin with. But once they are there, they might find it more productive and fulfilling to do something.
“Ah, the chicken and the egg problem, how depressing.”
Gotta start somewhere…
“Speaking of which, another problem with scientific blogging is that nobody is really around to finalize these posts. For example, this blog post has been edited a number of times after it was posted: minor errors were fixed and more content/links were added.”
Course, “live” articles can be a good thing, because they have a likelihood of being (more) accurate over an extended period (assuming there are committed people working on improving it, which I doubt). However, if other scholars were to reference to these articles to make their point, and the content has been changed, well that would create confusion and ruin the trustworthiness of it all. So I am sticking with my theory that digital open scientific commentary will work best for (pr)eprint platforms before they find root in the scientific blog concept, if ever.