Should PhD students do peer reviews?
It started out with a tweet. I simply wanted to figure out how many traditional peer reviews students did during their PhD, mostly out of sheer curiosity. Here are the results below:
How many formal peer reviews did you do/have you done during your PhD?
— Dr. Jon Tennant (@Protohedgehog) June 22, 2016
So one-third of students never reviewed anything, and around one-fifth did more than 5. That’s quite some disparity, and now I wanna figure out why. As Chris Jackson followed up, this survey could be a gateway to a cool discussion about how people feel about PhD students undertaking peer reviews.
Another aspect emerged too. Of those students who had done peer reviews, were they invited personally, or asked to by their supervisors for whatever reason (training, laziness). So many questions have emerged from this.
How does this vary across disciplines? In fields like Geology, it seems that PhD students under-taking reviews is rare, but in others it seems fairly common. But in Palaeontology, a fairly related discipline, it seems that students undertaking reviews might be much more common.
What is the reason for those zero counts? Is it because people don’t trust PhD students to reviews? Is it because they don’t get asked? Do students reject them because they feel under-qualified and that they lack sufficient expertise to do a ‘proper’ job?
And for those students who do review, do they go solo or get support from their supervisors or their lab group? Do they receive formal training, or just get chucked in the deep end?
Why do Editors invite PhD students at all? How do we decide who is qualified and who is not? Should there be standards to define these sorts of things? How does this shift depending on the publication record of students?
Now, this poll only achieved 296 votes. While that might be a lot compared to most polls, it definitely falls short of what we might call comprehensive.
So my follow up is this. Is there scope here to take this to the next level? Is there a point in taking it to the next level? What would such a survey look like? How would we distribute it? Would it lead to a publication?
If any of these questions above interest you, email me at jon.tennant.2@gmail.com and let’s get things rolling! 🙂
One thing to investigate is how number of publications might correlate with number of review invitations. After I got a publication or two under my belt, review invitations in the relevant areas started rolling in. Perhaps those with zero invites were still pretty new to the publication game?
(and answering your question, from an editor’s perspective, I often invite Ph.D. students _if_ they have a relevant publication on the topic — I do not think I have ever invited a student who hasn’t had a publication record)
That’s interesting. A really great add on to any research on this would be one from the Editor’s perspective. Or maybe both in one!
Yes, this would be an excellent thing to look at! I guess it would have to be differentiated between stage of PhD or completion too. Obviously we expect the number to accumulate through time and with publications, but would be great to parse apart those relationships. I wonder if n will be too small in many cases though.
I wonder about that, too. I wasn’t invited to peer review anything before my first first author paper. And that seems fair – like Andy pointed out from an editor’s perspective, as an author, I’ve never put someone in my cover letter who didn’t have a publication record on my topic.
I think those data will give us a much more complete picture, if they can be gotten. I’m not sure how to phrase that as a twitter poll … Maybe ‘My first invitation to peer review came after …’ with the options being a) After my first middle author paper, b) after my first first author paper c) after a conference talk d) other.
Yes, it seems like there is a minimum threshold of having at least n publications before being considered qualified enough to review. My experience was very weird happening before my first paper – apparently it was because Mike Benton recommended me to review as he knew about what I was working on! You can see the review here too: https://publons.com/publon/233044/ and I got Phil to look at it to make sure it was OK before submitting.
I was thinking about going beyond a Twitter poll for this. We could use SurveyMonkey or something to generate a really rigorous and comprehensive survey, and perhaps one for students and one for Editors.
Agree 100% with Andy here – I had about 5 papers published in rapid succession during the last semester & summer following my master’s program, and once these were out the review requests started trickling in. I suspect that there is going to be a strong relationship between getting something published and being asked to review – which is fair. Personally, knowing how much of a pain in the ass peer review can be, I’d rather have somebody review my work who is cognizant of peer review-induced bottom pain.
How would we go about tracking that potential relationship, do you think? It seems like it would be difficult to quantify. Perhaps the way April phrased it above would be the best way to figure it out.
Perhaps we need a system of joint-reviewership as a means of sort of training PhD students to be reviewers? A PhD is essentially an apprenticeship after all so why not treat the reviewing process the same?
A PI could be an invited reviewer and they could have the option of joint-reviewership whereby they enlist their student to co-review it but meanwhile the individual suggestions are tracked which would allow some compensation for the relative experience during the process?
Love that idea! So the questions would be on what scale is it necessary, who would deliver it, what would the training scheme look like etc. But I love the idea, and really think training things like this could be great. Perhaps a niche opportunity for a new business with publishers/editors as clients?
I’m not sure what it would mean in terms of it being a niche idea for a new business but I don’t think it would need be so formal? Merely just adding the option for co-reviewer surely?
Anecdote time!
My first review was before I’d had anything published… although this was because my supervisor (M) thought it would be good for me, and he held my hand through it. It was a review for somebody (K) I’d already met in person and corresponded with, and M had originally been invited to review K’s ms, but couldn’t because M and K were working on another paper together at the time, so there was a conflict of interest. I got invited by the journal editor at M’s suggestion, and M made sure he read my review before I sent it in. It was really good for me, and definitely not the same as having my supervisor foist a review onto me. Sounds a bit like Jon’s first review experience (you never forget your first time).
Since then, and since I’ve published my own stuff, I’ve been invited to review quite a bit more. It’s hard to tell how much of it is because a) I had a fairly widely read literature review paper on the topic last year, b) I’m one of the only people in my field who is active on twitter (i.e. uses it more than once every couple of weeks), or c) it’s a really small field, so there’s a limited pool of reviewers to choose from in the first place.
Interestingly, it’s stopped recently. I rebuffed a couple of closed publishers, saying that I’d love to review the paper because it looked fascinating, but only if it was made open (or if I got paid for it if it stayed closed). I don’t know if there’s a central blacklist of reviewers that editors/publishers circulate because it’s a waste of their time, but I haven’t been approached to review stuff for a while now, despite having a relatively big paper and a couple of conference talks.
I don’t think I’m very representative of other PhDs in my institute. I’m fairly sure that most PhDs here have never done a review.
In my experience, early career researchers can make excellent reviewers – often investing considerable time and effort in the process. As an editor for a journal, or if I have to decline a request to review, I’ll try and identify relevant ECRs to suggest, or approach; invariably this would be on the basis of having knowledge of a recent paper, or a recent conference presentation. My instinct is that it would be unusual to ask an individual formally to review a paper unless that individual already has experience of publishing (and of having read and acted on reviews).
Since reviewing is such a core part of the publishing process, we run a workshop on reviewing papers in our doctoral training programme.
Why not aim to have a combination of later, mid-, & early career (including PhD) reviewers? It would act as great training (especially if the peer review system is partially open, allowing reviewers to see each other’s anon feedback), and might also help reduce issues of unfairly biased reviews.