Well that was fun. Just spent two days sequestered with a group of people I don't know (okay, I know several of them, but not well − except Professor Rooster who, despite his name, is not annoying and always good fun). We've been reviewing grants. This is one of those terrible things we simply have to do. Okay, many people at my stage don't, but I have to. I'll tell you why in a bit.
For those who have never done this, the process goes like this (while I have done this in other countries, I'll describe the U.S. experience, which is the most ‘fun’). For some reason, this is not called a ‘grant review group,’ or any sort of ‘grant review thingy,’ instead it is called a ‘study section’. Nobody knows why. (Okay, somebody might, but I don't.) We each receive a stack (or ‘heaping pile’) of about 10−12 grants, vaguely related to our areas of expertise. Over the next few weeks, we read and evaluate them. Just before the actual meeting (for which we receive airfare, hotel and part of our additional costs), we get to read each other's reviews in preparation for the big day. On that day, we gather together (feeling terrific from our awesome flights in the back of the plane and our hotel that often has no hot water), say ‘hi’ for a few minutes, and then we start.
First we go through the rules, which are as follows:
1. Don't ever, ever say the ‘f’ word, which in this case is ‘funded’. I often say, the applicants are generally either ‘f’-ed or ‘f’-ed. Mostly, second ‘f’-ed.
2. Respect and reveal all conflicts of interest (COI). Except, of course, the fundamental COIs that relate to a) I also have a grant in this area, and I'm worried that supporting this grant might reduce enthusiasm for mine at some point; b) I depend on my grants to pay my salary and keep a roof over my head; c) if I set the bar too high, this will be applied to me next time and; d) if I set the bar too low, the other people around me won't think I'm a serious scientist. We'll come back to (d).
3. No knoodling. Technically, knoodling is a verb meaning, ‘talking to the person next to you’. Okay, usually it means ‘talking to the person next to you about something much more fun than what you're doing’, but you get the point.
4. This one is probably U.S. specific. We have to discuss ‘rigor’. We actually must use the word. If we don't use the word, they ask us to use the word. Apparently, this is a response to the concern, raised by people in the pharmaceutical industry, that funded research is not reproducible. (I've written about this before, see Replicant I, Replicant II and Replicant III. Get it? Replicant? Sorry, I crack myself up.) But somehow, we have decided that this is all about ensuring that all animal experiments are performed in both male and female animals. Don't get me wrong, I have nothing against that. But, lately, they are asking the reviewers if the applicants are analyzing the differences. I do have a problem with this. Sure, if there is a scientific justification for looking at sex differences, by all means do that. If not, why are you demanding it? But (I've asked) if you insist on it, then provide the funds to double all numbers − otherwise the experiments are not powered. Their response? Oh no, just do the independent analyses. So (I ask), you don't want us to be rigorous (if the experiment isn't powered, the results will not be significant − this is basic). So, in other words, just ensure that the applicants have used the words and actual rigor be damned.
Oh no, now I have to go off on the concept of ‘powering’ experiments. This is an important thing, especially when the experiment concerns people. How many must we recruit to a study in order to ensure that the effect we see is statistically significant? To answer this, we make a ‘guess’ as to what the effect will be and then determine how many subjects are needed for it to be significant if we're right. More recently, this has been extended to animal studies, and this is reasonable as well since we do not want to do experiments that show trends without significance. However, somehow, journals have taken to asking how our animal studies were powered − and here's the point: If the results are statistically significant then, indeed, our study is appropriately powered. Sometimes, I'm not sure that the editors who insist on this information understand this. Anyway, I digress. Back to the ‘rules’.
5. Be brief. We're not here to hear how smart you are. Nobody ever obeys rule No. 5.
I think there are more rules but, really, I don't ever listen. This is just the bit where we are racehorses being put into the gates, before the race starts. Then, we're off!
And this is where the problems start. Because, the fact is, the folks doing the reviewing are not necessarily the very best or the very brightest (don't get me wrong; many are, but not all). And, because of the very, very competitive nature of scientific funding, this can be disastrous. It's one of the corollaries of Murphy's law: If you add a teaspoon of wine to a barrel of sewage, you have sewage; if you add a teaspoon of sewage to a barrel of wine, you have sewage. And sadly, too many study sections are sewage, despite how much they may be composed of fine wine. (I know, pushing the metaphor here, but you know what I mean.)
So, finally, we are getting to the point. Many of you who read this may well be subjected to the vagaries of the grant review process, especially the not-so-vague part where the grant is not actually funded (and effectively, ‘f’-ed). It's good to know why.
First of all, perhaps the most important thing. Every single time a grant actually does well − not just ‘well’ but ‘good enough to be funded’, (and I have experienced many, many of these; maybe five or six each time I sit on a study section, which is often) − the discussions all have something in common: The first reviewer begins with a statement along the lines of “Okay, what they've found is really incredible!” Or, “This is just amazing.” Or, “This one just blew me away.” Get it? The reviewer is actually excited. Never, ever, did I hear of a grant being funded where the reviewer said, “The applicant proposes to follow up their previous publication with some good, steady work that will advance the field.” In fact, the majority of grants we see can be summarized in exactly this way − not bad, but not particularly exciting.
But that's never the reason applicants are given for why their work wasn't supported. Instead, we are pushed to say what is wrong with the application, so we adopt terminology that identifies problems: the application is diffuse, unfocused, represents marginal advances, lacks mechanism. Or, worst of all (for me): the application is too ambitious.
Now look, it is true that we can't support every good application − the funding agencies in most countries just don't provide enough to do that. And we, as reviewers, are not allowed to say that ‘this one just didn't float my boat’. But, usually, that's exactly the problem. It's important to keep this in mind. We can rail against this and shout, “Who are you to say my work isn't interesting?” But that won't do any good because, you see, they did not say ‘your work isn't interesting,’ they said, “The application is diffuse, unfocused, represents marginal advances and lacks mechanism, and − oh yes − it is too ambitious.” But what they meant was, ‘I just didn't think it was interesting’. See the problem?
Oh, actually, it can be even worse than this. In a research study performed a few years ago, hundreds of grant scores were compared against the time of day during which the review panel assessed the grant. There was a remarkable inverse correlation between the scores and the time from the previous meal. The researchers concluded that lower blood sugar was the most likely culprit − as glucose availability fell, so did the scores. Fortunately for those of us in the U.S., this problem has been ameliorated by discussing grants in the order of the preliminary scores, with the best scored grants being discussed right after breakfast (or lunch). Hopefully, the reviewers who provide the preliminary scores were well-fed before they sat down to reviewing your grant. But, sadly, this doesn't really solve all of the problems for us, the applicants.
So, when our grants are torpedoed (and believe me, it happens to all of us), do what I do: scream, pitch a fit, roll around on the ground, cry, call the reviewers every name I can think of, and then, when I've rested a bit, do it all again. Then, sometime later, I take a look at my grant and see if they have a point − maybe I didn't make the proposal as interesting as I could have (or maybe it just would never be that interesting and I need to rethink the whole thing).
There's much more to say but, right now I need a drink. You see, I was just sequestered for two days…
(to be continued)
