Anglophone versus Asian Peer Reviewers
For those active in the field of publishing and scholarly communication in general, you might have heard/read about this survey by Mark Ware & Mike Monkman, Mark Ware Consulting: Peer Review in Scholarly Journals – perspective of the scholarly community: an international study. I found it while going through the American Scientist Open Access Forum. I downloaded and glanced over the summary first because I am a busy man and all…
And I have to admit, I was not too excited about it at first, because the summarized results mostly conformed with other literature on the peer review concept. So I simply put it away in my peer review sources folder and told myself to look at it at some other time. Well, I guess that time is now and, on closer inspection, and with that I mean I downloaded the whole thing, I found the details in the full report to be much more interesting. One particular interesting aspect to me is the difference in responses between Anglophone and Asian respondents. More precisely, check out the following excerpts:
1. Peer review is widely supported. The overwhelming majority (93%) disagree that peer review is unnecessary.
2. Peer review improves the quality of the published paper. Researchers overwhelmingly (90%) said the main area of effectiveness of peer review was in improving the quality of the published paper. In their own experience as authors, 89% said that peer review had improved their last published paper, both in terms of the language or presentation but also in terms of correcting scientific errors.
3. There is a desire for improvement. While the majority (64%) of academics declared themselves satisfied with the current system of peer review used by journals (and just 12% dissatisfied), they were divided on whether the current system is the best that can be achieved, with 36% disagreeing and 32% agreeing. There was a very similar division on whether peer review needs a complete overhaul.
Nothing particularly surprising about that I should think, but the following excerpt (page 18):
Given the generally low level of overall dissatisfaction with peer review, though, it is perhaps surprising that a strong statement like “peer review in journals needs a complete overhaul” did not receive more disagreement – in fact respondents were divided, with 35% disagreeing versus 32% agreeing. There were clear regional differences on these questions, with Anglophones expressing net disagreement (43% opposed versus 27% supporting), while Asian respondents expressed net agreement (47% supporting versus 23% opposed), with Europe/Middle East/Row lying between these extremes.
I must say I am a bit surprised by this result. Why are Asian researchers more open to a complete overhaul of the journal peer review system than their Anglophone counterparts? Is the Asian journal peer review system in general noticeably different? Different enough to warrant this view of a complete overhaul? All of my sources on scholarly communication/peer review so far have been English and I am pretty sure I have never read anything about this distinction before (which makes these results that much more interesting to me).
And it becomes even weirder, on page 21 there is a section called ‘Regional differences on attitudes to peer review’. It lists the following results:
• in terms of overall satisfaction (Q3), Asian were slightly more satisfied than Anglophone respondents;
• but looking at Q4, Asian respondents were more likely to support critical statements about peer review, such as their net agreement for “peer review needs a complete overhaul” or “peer review is holding back scientific communication” compared to net opposition in other regions.
• On the question of the effectiveness, though, Asian respondents were more likely to agree that peer review was effective, especially regarding the detection of academic fraud and plagiarism.
In its defence, it was also prefaced with this piece:
There are substantial regional differences expressed, primarily between the Anglophone and Asian regions, on the questions of overall satisfaction (Q3), statements about the need for reform etc. (Q4) and the effectiveness of peer review (Q5). These differences are somewhat hard to understand, as they appear contradictory:
I say! What is going on? I do not get it. I suppose they do not have to be contradicting per se, but I agree they are odd stances to have. And since it was also stated that these results were somewhat hard to understand, I do not understand why it has not been mentioned in their summary. But let us focus on another important difference here: Why do Asian peer reviewers experience better results in detecting academic fraud and plagiarism? Are they doing something special that allows them to detect these things at a higher success rate? Do they have more advanced tools/techniques at their disposal to aid them with this? Are Asian institutions tougher on standards where successful frauds are harder to achieve and innocent mistakes harder to lose track of? Do Asian editors/peer reviewers have a more forceful “this could very well be a fraud/mistake” starting stance as opposed to “peer review is about trusting that the authors are honest with their reports”? “Maybe Asians researchers do more of this unethical stuff and are therefore more prone to be found out proportionally (but not necessarily in a higher ratio)?” Regardless, these are things we need to find out for the sake of potentially improving scholarly communication.
Ah well, let us save this for later and continue with this:
15. The average review takes 5 hours and is completed in 3-4 weeks. Reviewers say that they took about 24 days (elapsed time) to complete their last review, with 85% reporting that they took 30 days or less. They spent a median 5 hours (mean 9 hours) per review.
Good to have a number of hours specifically mentioned, it gives a better view of the workload of peer reviews. Another interesting tidbit concerning regional differences (page 42):
Asian and Rest of world respondents reported times over twice as long (13.4 and 12.5 hours respectively) as Anglophone reviewers (5.6 hours).
“I can see why you would find this stuff interesting, a lil bit controversial?”
Now, I personally would be very intrigued to know why there is such a difference between Anglophone and Asian respondents. I am assuming they are talking about peer reviewing scientific works in the same language, so it cannot be a case of language barrier. Why do Asian peer reviewers spend more time peer reviewing? Do Asian editors have a harder time finding the right/available peer reviewers for these manuscripts? Do these scientific works generally have more content? Are Asian peer reviewers more dedicated to get the best (or worst) out of a manuscript? And is that related to their experience that peer review can detect fraud and plagiarism better? “Maybe the manuscripts are generally of lower quality, which means more work for the peer reviewers? Or Asian researchers have more time to spend on peer reviewing?”
Moving on, here is an excerpt from the same page (42):
Comparing responses by the impact factor of the journal for which this review was completed showed that reviewers for high impact factor journals spent much less time on the review, 7 hours, than did reviewers for low impact factor journals (12 hours).
“Aha, my theory is becoming more likely now!”
Not necessarily, truth is, there is geographical bias in peer review/ journal impact factors. And the reason is simple: most of the world reads scientific literature in English. So when you write stuff not in English, it will be more difficult, if not impossible, to reach a large(r) audience. Impact factors of non english journals are therefore generally lower, while the quality does not necessarily, or at all, have to be that much lower as impact factors seem to indicate. Extreme case in point: Einstein was German. Had he written his stuff in German, I doubt it would have been considered of lesser quality. “But only with all other things being equal, you mean. Like (not) being able to communicate his stuff effectively!” I guess there is that, but that still does not make the content itself of lesser quality. Writing “De Aarde is bolvormig” is no less accurate than writing “The Earth is spherical”. Yet, the first sentence would be understood by fewer people (without translation) as opposed to the latter sentence.
However, this is no longer true when we are talking about the same languages here. In that case, one reason that could explain this is that the higher the journal impact factor, the better the editors/ peer reviewers. Which means there are time saving processes on two elements: editors screening and only letting through the better papers for peer reviews (and my theory is that the better the paper, the easier it is to peer review, because there is less room for errors/ improvements to spend the peer reviewers’ time on) and then there are the better peer reviewers which means they can more effectively scan for errors/improvements and point them out.
“Hmm, not that controversial, moving on!”
Asian respondents were more likely
thethan Anglophones to agree to the self-interested reasons (e.g. 33% of Asians supported “increase the chance of being offered a role in the journal’s editorial team” compared to 16% of Anglophones, and 53% agreed that “to enhance your reputation or further your career” was a reason for reviewing, compared to 42% of Anglophones).
“Aha! Asians are more honest? Or more greedy? Both?”
It could be a cultural thing. Maybe Asians are more competitive by nature? Or because there are less seats to fill? More financial pressure to get a high profile job? “The power of
the Asian Parent Syndrome sincere encouragement by their loved ones? “ Well, it could be anything, not very interesting to me I guess. Moving on…
As with reviewers, editors based in Anglophone regions handled more papers than those in other regions, especially Asia and Australasia.
Hmm, more hours per review, fewer number of reviews. Somewhat balanced, but not sure which to prefer, though.
The most common form of feedback to reviewers (used by 58% of editors) is to let them know the publication outcome (Q48).
Regionally, Anglophone and Australasian editors were more likely to give publication outcomes than those from Asia or Rest of world. Asia/Rest of world editors were more likely to give feedback on quality of report than average, and European editors less likely.
Again, why this difference? Do Asian editors have more time to give this feedback? Or do they feel it is necessary to help the peer reviewers improve regardless of time? I do not have any experience, but I think receiving feedback on their perceived quality of the peer review is great. Peer reviewers know how other professionals look at their evaluation skills, which should be important to peer reviewers.
Well, I guess I will end this with a nod to Open Access (or not, depending on how you look at it):
There was a fairly predictable distribution of responses by geographic region, with USA/Canada, Anglophone and Australasia groups reporting about 85% good or excellent access, dropping to 66% for Europe/M.East, 56% for Asia, and 53% for Rest of world (see graph on following page).
Could this be linked to all the questions I have addressed in this post? Who knows. It is reasonable to suggest that the more access people have to scientific literature, the more productive they can be/are in terms of carrying out scientific activities, including doing peer reviews. On the other hand, it seems to be inconsistent with the Asian experience concerning that ‘peer review was effective, especially regarding the detection of academic fraud and plagiarism’. It remains a mind-boggling experience…