Evaluate, evaluate, evaluate

© www.CartoonStock.com

When I was starting out on a doctorate, I’d look at the senior people in my field and wonder if I’d ever be like them. It must be great, I thought, to reach the advanced age of 40. By then you’d have learned everything you needed to know to do great science, and you could just focus on doing it. I suspect today’s crop of grad students are a bit more savvy than I was, but all the same, I wonder if they realise just how wrong that picture is – for two reasons.

First, you never stop learning. The field moves on. Instead of getting easier, it gets harder. I remember when techniques such as functional brain imaging first came along. The most competent people in that area were either those who had developed the methods, or young people who learned them as grad students. If you were of the generation above, you had three choices: ignore the methods, spend time learning them, or hire junior people who knew what they were doing. As the methods evolve, they get ever more complex, and meanwhile, your own brain starts to shrink. So if you are anticipating making it to a tenured post and then settling down in your armchair, think again.

Second, the more senior you get, the more of your time is spent, not on doing your own research, but on evaluation. You learn that an email entitled ‘invitation’ should not make your spirits rise: it’s just a desperate attempt to put a positive spin on a request for you to do more work for no reward. You get regular ‘invitations’ to review papers and grants, write job references, appraise promotion bids, sit on interview panels and examine theses. If you are involved in teaching, you’ll also be engaged in numerous other forms of appraisal.

I was prompted to think about this when someone asked on an electronic forum what was a reasonable number of doctoral theses to examine each year. The general consensus was two: though it will obviously depend on what other commitments someone has. It also varies from country to country. There are some jolly places in Europe where a PhD viva is just an excuse for a boozy party with a lot of dressing up in funny gowns and hats. In UK psychology, the whole thing is no fun at all: you have to read a document of 50,000-70,000 words reporting a body of work based on a series of experimental studies. You then write a report on it and see the candidate for a face-to-face viva, which is typically 2 to 3 hours long. Although failure is uncommon, it is not assumed that the candidate will pass (unlike in the viva-as-party countries), and weeping or catatonic candidates are not unheard of. Taking into account travel, etc., if you are going to do a proper job, you are probably talking about three days’ work. For this you get paid around the minimum wage – the fee for examining is typically somewhere between £120 and £200.

So why do we do it? The major reason is because the entire academic enterprise depends on reciprocity: we want people to examine our students and review our papers and grants. In addition, it’s important to maintain standards, and to ensure that degrees, promotions, publications and grants go to those who merit them. But the demands keep growing. In the 37 weeks of this year I’ve been asked to review 76 papers and six grants. I agreed to review 16 papers and three of the grants. This, of course, is nothing compared with being a journal editor or serving on a grants board, something that most of us will do at some point.

Clearly, if I agreed to do everything I was asked, I’d have no time for anything else. Of course, one learns to say no. But awareness of these pressures has made me look with rather a critical eye at how we use evaluation. There is, for instance, research suggesting that job interviews aren’t very useful at identifying good candidates:  we tend to be seduced by immediate impressions, which may not be a good indicator of a person’s suitability. Like most people, I’d be reluctant to take on an employee I hadn’t interviewed, but if Daniel Kahneman is to be believed, this is just because I am a victim of the Illusion of Validity.

I’m a supporter of the peer review system used by journals, and here I feel  I’m on more solid ground, because I can point to instances where my papers have been improved by input from reviewers. Nevertheless, where reviewing is used simply to reject/accept papers or grant proposals,  and where fine-grained decisions have to be made between many high-quality submissions, agreement between experts may be little better than chance (e.g. Fogelholm et al, 2012). Nevertheless, we stick with it, because it’s hard to know what to put in its place.

I’ve written a fair bit about that expensive and time-consuming evaluation process that UK academics engage in, the REF. It requires experts to make judgements of whether, for instance, papers are of 3* or 4* quality, a distinction based on whether the research is “world leading” or “internationally excellent…. but falls short of the highest standards of excellence.” The reliability of such judgements has not, to my knowledge, been evaluated, yet large amounts of funding depend on them. Those on REF committees are in the same situation as Pavlov’s poor dogs, having to make distinctions that are on the one hand impossible (discriminating circles and ellipses that become increasingly similar) and on the other hand very important (get it wrong and you get a shock).

There is one good thing about doing so much evaluation. You have the opportunity to see what others are doing – you may be the first person to read an important new paper, or examine a ground-breaking thesis. You may be forced to engage with different ways of thinking, and confronted with new topics and ideas. You may be able to provide useful input to authors. And since you yourself will be evaluated, it can be useful to see life from the other side of the table, as the person doing the evaluating. But all too often, even these advantages fail to compensate for the fact that as a senior academic you will spend more and more time on evaluation of others and less and less doing your own research.

Reference
Fogelholm, Mikael, Leppinen, Saara, Auvinen, Anssi, Raitanen, Jani, Nuutinen, Anu, & Väänänen, Kalervo (2012). Panel discussion does not improve reliability of peer review for medical research grant proposals Journal of Clinical Epidemiology, 65 (1), 47-52 DOI: 10.1016/j.jclinepi.2011.05.001

Discussion meeting vs conference: in praise of slower science

Pompeii mosaic
Plato conversing with his students

As time goes by, I am increasingly unable to enjoy big conferences. I'm not sure how much it's a change in me or a change in conferences, but my attention span shrivels after the first few talks. I don't think I'm alone. Look around any conference hall and everywhere you'll see people checking their email or texting. I usually end up thinking I'd be better off staying at home and just reading stuff.

All this made me start to wonder, what is the point of conferences?  Interaction should be the key thing that a conference can deliver. I have in the past worked in small departments, grotting away on my own without a single colleague who is interested in what I'm doing. In that situation, a conference can reinvigorate your interest in the field, by providing contact with like-minded people who share your particular obsession. And for early-career academics, it can be fascinating to see the big names in action. For me, some of the most memorable and informative experiences at conferences came in the discussion period. If X suggested an alternative interpretation of Y's data, how did Y respond: with good arguments or with evasive arrogance? And how about the time that Z noted important links between the findings of X and Y that nobody had previously been aware of, and the germ of an idea for a new experiment was born?

I think my growing disaffection with conferences is partly fuelled by a decline in the amount and standard of discussion at such events. There's always a lot to squeeze in, speakers will often over-run their allocated time, and in large meetings, meaningful discussion is hampered by the acoustic limitations of large auditoriums. And there's a psychological element too: many people dislike public discussion, and are reluctant to ask questions for fear of seeming rude or self-promotional (see comments on this blogpost for examples). Important debate between those doing cutting-edge work may take place at the conference, but it's more likely to involve a small group over dinner than those in the academic sessions.

Last week, the Royal Society provided the chance for me, together with Karalyn Patterson and Kate Nation, to try a couple of different formats that aimed to restore the role of discussion in academic meetings. Our goal was to bring together researchers from two fields that were related but seldom made contact: acquired and developmental language disorders. Methods and theories in these areas have evolved quite separately, even though the phenomena they deal with overlap substantially.

The Royal Society asks for meeting proposals twice a year, and we were amazed when they not only approved our proposal, but suggested we should have both a Discussion Meeting at the Royal Society in London, and a smaller Satellite meeting at their conference centre at Chicheley Hall in the Buckinghamshire countryside.

We wanted to stimulate discussion, but were aware that if we just had a series of talks by speakers from the two areas, they would probably continue as parallel, non-overlapping streams. So we gave them explicit instructions to interact. For the Discussion meeting, we paired up speakers who worked on similar topics with adults or children, and encouraged them to share their paper with their "buddy" before the meeting. They were asked to devote the last 5-10 minutes of their talk to considering the implications of their buddy's work for their own area. We clearly invited the right people, because the speakers rose to this challenge magnificently. They also were remarkable in all keeping to their allotted 30 minutes, allowing adequate time for discussion. And the discussion really did work: people seemed genuinely fired up to talk about the implications of the work, and the links between speakers, rather than scoring points off each other.

After two days in London, a smaller group of us, feeling rather like a school party, were wafted off to Chicheley in a special Royal Society bus. Here we were going to be even more experimental in our format. We wanted to focus more on early-career scientists, and thanks to generous funding from the Experimental Psychology Society, we were able to include a group of postgrads and postdocs. The programme for the meeting was completely open-ended. Apart from a scheduled poster session, giving the younger people a chance to present their work, we planned two full days of nothing but discussion. Session 1 was the only one with a clear agenda: it was devoted to deciding what we wanted to talk about.

We were pretty nervous about this: it could have been a disaster. What if everyone ran out of things to say and got bored? What if one or two loud-mouths dominated the discussion? Or maybe most people would retire to their rooms and look at email. In fact, the feedback we've had concurs with our own impressions that it worked brilliantly. There were a few things that helped make it a success.

• The setting, provided by the Royal Society, was perfect. Chicheley Hall is a beautiful stately home in the middle of nowhere. There were no distractions, and no chance of popping out to do a bit of shopping. The meeting spaces were far more conducive to discussion than a traditional lecture theatre.
• The topic, looking for shared points of interest in two different research fields, encouraged a collaborative spirit, rather than competition.
• The people were the right mix. We'd thought quite carefully about who to invite; we'd gone for senior people whose natural talkativeness was powered by enthusiasm rather than self-importance. People had complementary areas of expertise, and everyone, however senior, came away feeling they'd learned something.
• Early-career scientists were selected from those applying, on the basis that their supervisor indicated they had the skills to participate fully in the experience. Nine of them were selected as rapporteurs, and were required to take notes in a break-out session, and then condense 90 minutes of discussion into a 15-minute summary for the whole group.  All nine were quite simply magnificent in this role, and surpassed our expectations. The idea of rapporteurs was, by the way, stimulated by experience at Dahlem conferences, which pioneered discussion-based meetings, and subsequent Strüngmann forums, which continue the tradition.
• Kate Nation noted that at the London meeting, the discussion had been lively and enjoyable, but largely excluded younger scientists. She suggested that for our discussions at Chicheley, nobody over the age of 40 should be allowed to talk for the first 10 minutes. The Nation Rule proved highly effective - occasionally broken, but greatly appreciated by several of the early career scientists, who told us that they would not have spoken out so much without this encouragement.

I was intrigued to hear from Uta Frith that there is a Slow Science movement, and I felt the whole experience fitted with their ethos: encouraging people to think about science rather than frenetically rushing on to the next thing. Commentary on this has focused mainly on the day-to-day activities of scientists and publication practices (Lutz, 2012). I haven't seen anything specifically about conferences from the Slow Science movement (and since they seem uninterested in social media, it's hard to find out much about them!), but I hope that we'll see more meetings like this, where we all have time to pause, ponder and discuss ideas.  

Reference
Lutz, J. (2012). Slow science Nature Chemistry, 4 (8), 588-589 DOI: 10.1038/nchem.1415

The academic backlog

Photo from http://www.pa-legion.com

Here’s an interesting question to ask any scientist: If you were to receive no more research funding, and just focus on writing up the data you have, how long would it take? The answer tends to go up with seniority, but a typical answer is 3 to 5 years.

I don’t have any hard data on this – just my own experience and that of colleagues – and I suspect it varies from discipline to discipline. But my impression is that people generally agree that the academic backlog is a real phenomenon, but they disagree on whether it matters.

One view is that completed but unpublished research is not important, because there’s a kind of “survival of the fittest” of results. You focus on the most interesting and novel findings, and forget about the rest. It’s true that we’ve all done failed studies with inconclusive results, and it would be foolish trying to turn such sow’s ears into silk purses.  But I suspect there’s a large swathe of research that doesn’t fall into that category, but still never gets written up.  Is that right, given the time and money that have been expended in gathering data? Indeed, in clinical fields, it’s not only researchers putting effort into the research – there are also human participants who typically volunteer for studies on the assumption that the research will be published.

I’m not talking about research that fails to get published because it’s rejected by journal editors, but rather about studies that don’t get to the point of being written up for publication. Interest in this topic has been stimulated by Ben Goldacre’s book Bad Pharma, which has highlighted the numerous clinical trials that go unreported – often because they have negative results. In that case the concern is that findings are suppressed because they conflict with the financial interests of those doing the research, and the Alltrials campaign is doing a sterling job to tackle that issue. But beyond the field of clinical trials, there’s still a backlog, even for those of us working in areas where financial interests are not an issue.

It’s worth pausing to consider why this is so. I think it’s all to do with the incentive structure of academia. If you want to make your way in the scientific world, there are two important things you have to do: get grant funding and publish papers. This creates an optimisation problem, because both of these activities take time, and time is in short supply for the average academic.  It’s impossible to say how long it takes to write a paper, because it will depend on the complexity of the data, and will vary from one subject area to the next, but it’s not something that should be rushed. A good scientist checks everything thoroughly, thinks hard about alternative interpretations of results, and relates findings to the existing research literature. But if you take too much time, you’re at risk of being seen as unproductive, especially if you aren’t bringing in grant income. So you have to apply for grants, and having done so, you have then to do the research that you said you’d do. You may also be under pressure to apply for grants to keep your research group going, or to fund your own salary.

When I started in research, a junior person would be happy to have one grant, but that was before the REF. Nowadays  heads of department  will encourage their staff to apply for numerous grants, and it’s commonplace for senior investigators have several active grants, with estimates of around 1-2 hours per week spent on each one. Of course, time isn’t neatly divided up, and it’s more likely that the investigator will get the project up and running and then delegate it to junior staff, then putting in additional hours at the end of the project when it’s time to analyse and write up the data. The bulk of the day-to-day work will be done by postdocs or graduate students, and it can be a good training opportunity for them. All the same, it’s often the case that the amount of time specified by senior investigators is absurdly unrealistic. Yet this approach is encouraged: I doubt anyone ever questions a senior investigator’s time commitment when evaluating a grant, few funding bodies check whether you’ve done what you said you’d do, and even if they do, I’ve never heard of a funder demanding that a previous project be written up before they’ll consider a new funding application.

I don’t think the research community is particularly happy about this: many people have a sense of guilt at the backlog, but they feel they have no option. So the current system creates stress as well as inefficiency and waste. I’m not sure what the solution is, but I think this is something that research funders should start thinking about. We need to change the incentives to allow people time to think. I don’t believe anyone goes into science because they want to become rich and famous: we go into it because we are excited by ideas and want to discover new things. But just as bankers seem to get into a spiral of greed whereby they want higher and higher bonuses, it’s easy to get swept up in the need to prove yourself by getting more and more grants, and to lose sight of the whole purpose of the exercise – which should be to do good, thoughtful science.  We won’t get the right people staying in the field if we value people solely in terms of research income, rather than in terms of whether they use that income efficiently and effectively.