graphic

I'm putting on my top hat, tying up my white tie, brushing up my tails – or at least finding a clean t-shirt. Going out in style. Because I've just been invited – I'm so excited, I can already taste the toast – I'll tell you where. It's...um...oh.

Strike that. The invitation was to review a rather large grant on a rather esoteric topic by someone I don't know, for a funding organization I've never heard of, in a country I've never been to. And it's not a visit – I'm to send the review by e-mail. Wait just a moment while I tell them that I've got a previous appointment to wash my hair.

I've been receiving a lot of invitations just like this recently. Invitations to review grants and papers. The e-mail always says something like this: `You are invited to review MS#33701 for the J. Mssv. Time Cmtmt', and contains a directive that I am not to respond to the e-mail but instead to go to their link (click below) and respond online. If I don't, or if I respond directly to the e-mail, I'll receive the same e-mail every few days with computer-generated messages of how difficult I'm being.

Don't get me wrong – I review a lot. I think that it is part of our job to ensure that science moves forward through the essential process of peer review. It's important, because without it we would have to leave the decision making to others who may not be as expert as we are and then we'd complain about the results. And I depend on other experts to review and critique my work, so that I'll know why we spent so much time on something that nevertheless crashed and burned. It's how science is done, and while I'm not complaining...okay, I am complaining, but not about that.

Once upon a time, a long, long time ago, maybe even more than a year ago, requests to review came from people. They would request that we find time in our busy schedules to critique a grant or manuscript and respond to us if we asked questions about it. We might need more information or ask for more time or offer an apology if we couldn't do it this time. And we'd get a polite response. There was an veneer of sophistication about the whole thing that made it if not enjoyable (it never was, not even once upon a time) at least not onerous.

That was before the process was taken over by robots.

Alan Turing, one of the great minds of the last century, who not only broke Enigma but also invented the computer (so that he could break Enigma and e-mail his friends), proposed the idea of artificial intelligence, or AI. He predicted that AI would become a goal of a new field of science (but didn't predict that it would also be an awful Spielberg effort – he was a genius, but not that smart) that would try to create machines that think. The problem was how would we know that the machines think? Maybe they were faking it.

This was a real problem, because we already knew about Clever Hans. Clever Hans (no, there aren't any bad movies about him yet, but wait) was a horse who astounded the world by doing arithmetic. He was demonstrably a thinking horse. It turned out that CH (Turing's nickname for him) was taking cues from his owner to get the answers. He only seemed to be doing arithmetic. The same could be said for a computer that could do sums, break codes or play extremely good chess. They can do it, but are they intelligent?

Turing therefore proposed the Turing Test: he proposed that a machine would be intelligent when a person communicating with it (say on a computer terminal) couldn't tell if it was a person or not. If it seemed to be intelligent, and you couldn't tell if it wasn't, then it was. And for many years after that, AI researchers used this as the gold standard (until there wasn't a gold standard anymore).

Now, all these years later, I find that I can't tell when I'm being talked to by a machine. I receive these requests, these invitations, which were presumably initiated by a person but may or may not be any longer under anyone's control. How often have I responded, perhaps grumpily, to an e-mail informing me that my review is late (when it wasn't or was only a bit late), only to be told subsequently that the machine had sent the message in error? How do I know that this subsequent message wasn't generated by the same machine?

As the publication of science accelerates, and it is accelerating (once, it was possible to have an entire year of your favorite journal editions on the shelf above your desk), the demand on publishers, editors, and reviewers has increased to the point that the system is automated. But how much? We submit a paper and wait for reviews, and when they come we respond, and a decision is made. In the cases of the best journals, we know (or hope) that intelligent editors stand between our efforts and those of the reviewers to reach a decision. But always? It does seem that in too many cases the role of the journal is to provide an automated system that serves to preserve anonymity of the reviewers, who make the decision by inputting numbers (`How would you rate this paper on a scale of 1-10?'). The response from the journal is in the form of a standard letter, and we have no one to talk to. Hello? Is anyone out there?

I have a fantasy that, in the interest of financial responsibility, these journals have replaced their editors with machines. It was these same machines that came up with the idea that if people insist on being reviewed by other people, then that was a privilege, not only to the authors but to the reviewers, who are just as dispensable as the editors. They aren't even sure that authors are that important, because they generally write pretty poorly and often can't agree on the facts anyway. But since they are in the business of serving, because of their programming (or at least their prime directive) they'll continue to invite humans to participate. Don't forget To Serve Man was a cookbook.

Reviewing is a hard job, one that takes many hours away from the things we need or want to do, but one that we do in the interest of making science better. It has to be done with the knowledge that it helps, and it should be done with rigor and care. It's vital.

I don't mind saving some time by using machines to organize the process. I don't even mind having to go a bit out of my way to identify an entity in the system that can make a decision based on the available information, provided it is done with the same care and rigor that goes into doing research, writing papers and grants, and reviewing them.

But maybe all I'm asking is to be asked. Please don't invite me to give up my weekend to instead sit at a desk critiquing something that I probably wouldn't have chosen to read. If you do, I'll be washing my cat.

And I don't even have a cat.