The Green Paper’s level playing field risks becoming a morass

© Cartoonstock

The Green Paper Fulfilling our Potential: Teaching Excellence, Social Mobility and Student Choice is a consultation document by the Department of Business, Innovation and Skills (BIS) that sets out plans for a radical shake-up of English universities. One of its many goals is to encourage ‘alternative providers’, aka private higher education institutions (HEIs), into the system.

The rationale behind this move has four elements:

Widening access: The need to widen access to higher education is a central plank of the Green Paper. New providers are seen as part of this: they “can offer programmes that are attractive to hard-to-reach communities and to groups of people that are not currently well-served.”

Equipping students for the world of work:  As I have argued elsewhere, the Green Paper is rather disingenuous in creating the impression that businesses are unhappy with the quality of university graduates.  Nevertheless, there is a shortage of STEM graduates and there is scope for different approaches, one of which is the ‘Degree Apprenticeship’, announced by BIS in March 2015, where the student “will be employed throughout and so have the opportunity to develop employability skills that employers value.” One can see that these apprenticeships would be appealing to those on a low income (and to many others as well!) as the cost of course fees is shared between government and employers.

Competition: For a conservative government, competition is a key driver for improving the world. We are told that: “Widening the range of high quality higher education providers stimulates competition and innovation, increases choice for students, and can help to deliver better value for money.”

Fairness:  Since students are now paying full fees, it seems unfair to allow traditional universities to have a monopoly on the higher education market. According to the Green Paper:  “More providers entered the sector in the last five years than at any time since the last major expansion in 1992, but it’s still too difficult to set up a new institution. We want to see a level playing field for all providers and a faster route to becoming a university.”  

I knew very little about the “alternative providers” that BIS is so keen to encourage, so I had a dig around on the internet. I found a HEFCE Register of Providers but this contained only 12 institutions with degree-awarding powers, and it did not have historical data. I also wondered about accuracy, as the register did not include the New College of the Humanities*. It did include an assortment of religious institutions (e.g., Assemblies of God Incorporated, Elim Foursquare Gospel Alliance, Salvation Army) and those offering training in various therapies (British School of Osteopathy, the College of Integrated Chinese Medicine), but of these only the British School of Osteopathy was a recognised HEI with degree-awarding powers. An email to HEFCE confirmed that the register was set up only in 2014. They gave me a contact for the department of Business, Innovation and Skills (BIS), but an email to them requesting a source for the claim “More providers entered the sector in the last five years than at any time since the last major expansion in 1992” has so far not elicited a response.

Further digging around established that in January 2015, plans were announced to form an Independent Universities Group (IUG) representing private institutions with degree-awarding powers or university title. The goal was to establish a kind of “Russell Group” of independent providers, who could be clearly distinguished from what the Times Higher Education called “dodgy for-profit colleges”. However, discussions between these institutions seem to have been derailed by the election and the IUG has not so far been formed.

I looked at the courses on offer at this “high-quality” end of the sector, to see how far they meet the four goals outlined above.

• The website of Ashridge Business School takes you to Hult International Business School, a global organisation with campuses around the world. They offer one undergraduate degree, a Bachelor of Business Administration, for a tuition fee of £23,580 per annum.
• BPP University (owned by the US Apollo Group) offers undergraduate courses in Accountancy and Finance, Law, Business and Management, and Health. “Health” may be something of a misnomer, given that it includes an “Integrated Undergraduate Masters in Chiropractic (MChiro)” (see this article). Fees for undergraduate degrees are £6K per annum for a 3-year course and £9K per annum for a 2-year course.
• The University of Buckingham was founded in 1973 and is the private university in the UK operating under a Royal Charter. It runs a range of degrees in arts and humanities, and offer a more intensive 2-year degree curriculum, at a fee of just over £9K per annum. In 2015, they expanded to offer a medical degree at a cost of just over £35K per annum for a 4.5 year course.
• The College of Estate Management offers part-time online courses in areas such as building surveying and estate management, at a cost of around £5.5K per annum.
• The University of Law (owned by Montagu Private Equity), as its name suggests, offers a range of undergraduate and postgraduate law courses, and charges £9K per annum for a 2-year undergraduate LLB or £6K per annum for a 3-year degree.
• Regent’s University London offers undergraduate degrees in a wide range of subjects in arts and humanities, charging just under £16K per annum.
• RDI (a subsidiary of US firm Capella Education Company) partners with various UK Universities to offer distance learning courses. You can, for instance, take a Psychology degree at Anglia Ruskin University through RDI. The nature of the course gives flexibility in duration of study, with a total cost of just over £12K.
• Richmond, the American International University in London, offers a range of undergraduate courses, mostly in business and finance, with an annual fee of £9K.

My overall impression is that these institutions introduce some innovative practices that could help achieve the goal of widening access. In particular, we see places offering shorter, more intensive degrees, part-time degrees, and distance learning. Some of them are competing with public universities in terms of cost, because there is currently a lower ceiling on allowable fees, but many of them offer a restricted range of courses. There is little evidence that these alternative providers will do a better job than other HEIs in catering to the needs of employers. STEM subjects are expensive to teach and are barely represented in courses offered by alternative providers. The medical degree offered by the University of Buckingham is an exception, but it is priced out of the reach of all but the most wealthy. Only one of the private institutions described above, BPP University, features on a list of providers of Degree Apprenticeships: in general, it is HEFCE-funded institutions that have introduced these apprenticeships.  

Of course, it could be argued that a more diverse set of alternative providers would be seen if we could free them from the regulatory barriers that the Green Paper complains of. However, the regulation is there for good reason. The fact that the IUG group want to distance themselves from others in this sector should sound a note of caution about the potential downsides of the alternative provider market. It is noteworthy that most of the alternative providers from the IUG list offer courses that are eligible for loans from Student Finance England.  Only a year ago, Christopher Banks, Chairman of the Quality Assurance Agency was complaining that money was being squandered on loans to students attending dubious for-profit colleges.  Some institutions would encourage students to take out loans to cover their fees, but then offer inadequate courses associated with high drop-out rates.  Mr Banks was quoted as saying: “I would like to make sure we quickly respond and reinforce the need for consistent quality in higher education, because there is a danger, otherwise, that [the growth of private providers] will tarnish the reputation of the sector.”

We can also learn lessons from the USA, where the behaviour of some for-profit colleges in recruiting students to pay for worthless degrees is nothing short of a scandal. We should not forget that, only four years ago, the Apollo Group, who own BPP University, was investigated by the US Government for misleading students.

It is telling that the Green Paper recognises the potential problems of a marketised higher education sector, noting that students need protection from the consequences when their institutions fail. This has not previously been recognised as a risk, for the good reason that English universities have not failed. More recently, we have seen considerable hardship inflicted on overseas students when institutions have had their licences to sponsor overseas students revoked. In future, we may find students suffering, even after completing their course, if their degree comes from an institution that no longer exists. It is frankly surprising that anyone should be talking about reducing regulation of the private higher education sector and speeding up approvals when there is already evidence of unprecedented risks associated with the entrance of new providers.

There are many students who might benefit from having a wider range of options in higher education; I doubt that private HEIs are going to ‘drive up standards’ through competition, but they could potentially make a difference by complementing what is currently on offer. But if experience here, and in the USA, has taught us anything, it is that, in order to work well, alternative providers need to be carefully regulated and accredited only after establishing a solid track record.  

*Correction: 24th December 2015; David Sweeney of HEFCE pointed out that the New College of Humanities is on the directory and can be found via the web interface. On the .csv file that I downloaded it is included under an alternative name, i.e. Tertiary Education Services Ltd.
PS, 3rd January 2016: The relationship between Tertiary Education Service and New College of the Humanities is discussed further here.

A lamentable performance by Jo Johnson

Last week I wrote a blogpost for the Council for Defence of British Universities, in which I discussed the government’s Green Paper “Fulfilling Our Potential”. The Green Paper is a consultation document that introduces, among other things, the Teaching Excellence Framework (TEF). This is an evaluation process for teaching that is intended to parallel the Research Excellence Framework (REF). I argued against it. I’m concerned that the imposition of another complex bureaucratic exercise will do damage to our Higher Education system, and I think that the case for introducing it has not been made. Among other things, I noted that there was little evidence for the claim that there was widespread dissatisfaction among students.  Put simply, my argument was, if it ain’t broke, don’t fix it.

A day after my blogpost appeared, there was a select committee meeting of the department of Business, Innovation and Skills to take oral evidence on topics relating to the Green Paper. The oral evidence is available here as a transcript. This is fascinating, because there appeared to be a difference of opinion between the Minister, Jo Johnson, and the others giving evidence in terms of their views of the state of teaching in our Universities. The most telling part of the session was when Jo Johnson was challenged on his previous use of the word ‘lamentable’ to describe teaching in parts of our higher education system. I am reproducing the transcript here in full, despite its length, as it is important context to what comes next:

Chair: Can I take you back to your speech on 9 September about higher education fulfilling our potential? There is a particular passage in there that is really interesting, talking about a family and varying levels of experience. May I quote you? “This patchiness in the student experience within and between institutions cannot continue. There is extraordinary teaching that deserves greater recognition. And there is lamentable teaching that must be driven out of our system.” Could you tell us where that lamentable teaching is? 
Joseph Johnson: Thank you very much for having me, and I will certainly come to that in just one second. What I want to say is that it is a pleasure to be here to give evidence before you, and I am delighted at the interest the Committee is taking in this very important subject. There is extraordinary excellence across our higher education system; that is the first thing to say. We have a great university system in this country, it is one of our national success stories, and it is a terrific calling card for us on the global stage. It is very important to put that frame in context out there, but of course the sector cannot stand still. University systems around the world are becoming more and more competitive. Developing countries are putting in place stronger and stronger frameworks for their own university systems, and in that environment it is incumbent on us to continue to make a great sector greater still. That is the opening frame of how I see the sector. It is continuing and continuous improvement, and that is all the more important for us, as a sector, at a time when we are seeing ever-increasing numbers of our young people go through university. We are now at a stage of mass higher education in this country, with about 47% of people likely to go through higher education at some point in their lives, and it is vital for us, as a Government, that we ensure that they are getting the best-quality experience for the time and for the money that they are investing in higher education.  You referred back to a speech I gave to Universities UK and I used that word; it made a point. It made a point that there is, essentially, patchiness in provision and I am happy, before you, to give evidence of where I see patchiness, if that is helpful.
Chair: Would you use the word “lamentable” again?
Joseph Johnson: I certainly made the point, and the point was made in order to highlight the fact that there is patchiness and variability in provision. 
Chair: “Patchiness” is not “lamentable” though. 
Joseph Johnson: Patchiness and variability are the features that I want to stress before you today. I am quite happy to give plenty of supporting evidence of that and I think the sector, in its responses to you as a Committee, has also agreed that there is a need to focus on the quality of teaching in our institutions. I am happy to give more evidence on that, if you want.
 Chair: I would be very keen for you to give evidence to us, but just to push you on this, “lamentable” is an extraordinarily strong word. Would you use it again? 
Joseph Johnson: I think there are patches of poor-quality provision and whether or not we want to use that word—
Chair: Lamentable patches?
Joseph Johnson: Whether we want to use that word, it certainly made a point. It highlighted the point I was trying to make. I do not see the need to repeat it ad nauseam, but I think I made my point.

Johnson clearly wanted to move away from discussions about his choice of words and onto the ‘evidence’. I’m going to focus here on what he said about results from the National Student Survey (NSS). There are many pertinent questions about how far the NSS can be taken as evidence of teaching quality, but I will leave those to one side and just focus on what the Minister said about it, which was:

In the NSS 2015 survey, two thirds of providers are performing well below their peers on at least one aspect of the student experience; and 44% of providers are performing well below their peers on at least one aspect of the teaching, assessment and feedback part of the student experience.

I was surprised by these numbers for two reasons: first, they seemed at odds with other reports about the NSS that had indicated a high level of student satisfaction. Second, they seemed statistically weird. How can you have a high proportion of providers doing very poorly without dragging down the average – which we know to be high? I looked in vain online for a report that might be the source of these figures. Meanwhile, I decided to look myself at the NSS 2015 results, which fortunately are available for download here.

All items in the NSS are rated from 1 (definitely disagree) to 5 (definitely agree). I focused on full-time courses, and combined all data from each institution, rather than breaking it down by course, and I excluded any institutions with fewer than 80 student responses, as estimates from such small numbers would be less reliable. Then, to familiarise myself with the data, and get an overall impression of findings, I plotted the distribution of ratings for the final overview item in the survey, i.e., “Overall, I am satisfied with the quality of the course”. As you can see in Figure 1, the overwhelming majority of students either ‘agree’ or ‘definitely agree’ with this statement. Few institutions get less than 75% approval, and none has high rates of disapproval.

Figure 1: Distribution of responses to item 22: "Overall I am satisfied with the quality of the course"

Johnson’s comments, however, concerned individual items on the survey.

As you can see in the table below, there is variation between items in ratings, with lower mean scores for those concerning feedback and smooth running of the course, but overall the means are at the positive end of the scale for all items.

Item	Mean (SD)
1. Staff are good at explaining things.	4.19 (0.11)
2. Staff have made the subject interesting.	4.12 (0.14)
3. Staff are enthusiastic about what they are teaching.	4.3 (0.14)
4. The course is intellectually stimulating.	4.19 (0.17)
5. The criteria used in marking have been clear in advance.	4.02 (0.19)
6. Assessment arrangements and marking have been fair.	4.01 (0.19)
7. Feedback on my work has been prompt.	3.79 (0.24)
8. I have received detailed comments on my work.	3.95 (0.23)
9. Feedback on my work has helped me clarify things I did not understand.	3.85 (0.21)
10. I have received sufficient advice and support with my studies.	4.09 (0.16)
11. I have been able to contact staff when I needed to.	4.27 (0.16)
12. Good advice was available when I needed to make study choices.	4.11 (0.15)
13. The timetable works efficiently as far as my activities are concerned.	4.09 (0.18)
14. Any changes in the course or teaching have been communicated effectively.	3.95 (0.24)
15. The course is well organised and is running smoothly.	3.87 (0.27)
16. The library resources and services are good enough for my needs.	4.19 (0.26)
17. I have been able to access general IT resources when I needed to.	4.28 (0.23)
18. I have been able to access specialised equipment, facilities or rooms when I needed to.	4.11 (0.23)
19. The course has helped me to present myself with confidence.	4.18 (0.13)
20. My communication skills have improved.	4.31 (0.13)
21. As a result of the course, I feel confident in tackling unfamiliar problems.	4.21 (0.12)
22. Overall, I am satisfied with the quality of the course	4.16 (0.18)

It could be argued that Johnson was quite right to focus not so much on the average or the best, but rather on the range of scores. However, the way he did this was strange, because he computed percentages of those who did poorly on any one of a raft of measures. This seems quite a high bar, as a low rating on a single item could create the impression of failure.

In order to reproduce Johnson’s figures, I had to work out what he meant when he said an institution performed “well below” its peers. I looked at two ways of computing this. First, I just considered how many institutions fell below an absolute cutoff on ratings: I picked out cases where there were 20% or more ratings in categories 1 (strongly disagree) or 2 (disagree); this was entirely arbitrary, and determined by my personal view that an institution where one in five students is dissatisfied might be looking to do something about this. Using this cutoff, I found that 24% of institutions did poorly on at least one item in the range 1-9 (covering teaching assessment and feedback), and 35% were rated poorly on at least one item from the full set of 22 items. This was about half the level of problems reported by Johnson.

I wondered whether Johnson had used a relative rather than absolute criterion for judging failure. The fact that he talked of providers performing ‘well below their peers’ suggested he might have done so. One way to make relative judgements is to use z-scores, i.e. for every item, you take the mean and standard deviation across all institutions and then compute a z-score which represents how far this institution scores above or below the average on that item. Using a cutoff of one standard deviation, I obtained numbers that looked more like those reported by Johnson – 43% doing poorly on at least one of the items in the range 1-9, and 59% doing poorly on at least one item from the entire set of 22. However, there is a fatal flaw to this method; unless the data have a strange distribution, the proportions scoring below a z-score cutoff are entirely predictable from the normal distribution: for a one SD cutoff, it will be around 16 per cent. You’d get that percentage, even if everyone was doing wonderfully, or everyone was doing very poorly, because you are not anchoring your criterion to any external reality. For anyone trained in statistics this is a trivial point, but to explain it for those who are not, just look again at Table 1. Take, for instance, item 21, where the mean rating is 4.21 and standard deviation 0.12. These scores are tightly packed and so a score of 4.09 is statistically unusual (one SD lower than average), but it would be harsh to regard it as evidence of poor performance, given that this is still well in the positive range.

I have no idea what method Johnson relied upon for the statistics he presented: I am trying to find out and if I do I will add the information to this post. But meanwhile, I have to say I find it disturbing that NSS data appear to have been spun to paint the state of university teaching in as bad a light as possible. We know that politicians spin things all the time, but it is a serious matter if a Government minister presents public data in a misleading way when giving evidence before a select committee. Those working in primary and secondary education, and in our hard-pressed health service, are already familiar with endless reorganisations that are justified by arguing that we ‘cannot stand still’ and must ‘remain competitive’. We are losing good teachers and doctors who have just had enough. We need to draw back from extending this approach to our Higher Education system. Of course, I am not saying it is perfect, and we need to be self-critical, but the imposition of yet another major shake-up, when we have a system that has an international reputation for excellence, would be immensely damaging, and could leave us with a shortage of the talent that universities depend upon.

NB. You can reproduce what I did by looking at this R script, where my analysis is documented. This has flexiblity to look at alternative ways of defining the key item in Johnson’s analysis, i.e. the definition of “well below one’s peers”.

PS 14th Dec 2015: Another source of evidence cited in the Green Paper is this report from HEPI. Well worth a read. Confirms widespread student satisfaction with courses. Does show that 'value for money' is rated much higher in Scotland (low fees) than England (£9K per annum) http://www.hepi.ac.uk/2015/06/04/2015-academic-experience-survey/ 

PS. 16th Dec 2015. I have now had a response from BIS. It is rather hard to follow, but indicates that they do use a relative rather than absolute criterion for expected scores. Expected scores are also benchmarked to take into account student characteristics. I am currently struggling to understand how 66% of institutions can score more than 3 SD below a benchmark on at least one item, given that a z-score as extreme as -3 is expected for only 0.1% of a population. When I get the opportunity, I will look at the HEFCE source they recommend to see if it offers any enlightenment. 

Here is the BIS response:

In the NSS 2015 survey, two thirds of providers are performing well below their peers on at least one aspect of the student experience;

The statistic is based on the National Student Survey 2015, including HEFCE funded institutions with undergraduate students (123 institutions). Answers to all questions (Q1-22) are then compared to their institutional benchmarks. Those institutions that are statistically significantly below their benchmark for at least one question are counted (77 in 2015 NSS data). Therefore, 63% of institutions are performing below their benchmarks on one aspect of the student experience in 2015.

44% of providers are performing well below their peers on at least one aspect of the teaching, assessment and feedback part of the student experience.

This statistic is calculated using the same method as above. The difference is that it is based on Q1-9 of the NSS survey; where Q1-4 relate to teaching and Q5-9 relate to assessment and feedback.

Benchmarks

Benchmarks are the expected scores for each question for an institution given the characteristics of its students and its entry qualifications. Benchmarks are based on initial calculations by HEFCE. More information can be found on their website, where benchmarks for Q22 are published.

Statistical significance

Scores are considered statistically different from their benchmarks if they are more than 3 standard deviations and 3 percentage points below their benchmarks. This is the same convention used in the UK HE performance indicators.

I had previously contacted HEFCE who explained they had not been involved in generating the figures reported by BIS and suggested I contact BIS directly for information They also said:

As you will be aware HEFCE currently publishes benchmark data for question 22 of the NSS only and the current published data based on this question shows a relatively small proportion of institutions who are significantly below their benchmark. (The data can be accessed from www.hefce.ac.uk/lt/nss/results/2015)

We together with the other UK funding bodies have highlighted our interest in developing benchmarks for other questions in the recent consultation on information about learning and teaching, and the student experience, however this would need to be considered in a thorough and robust manner including any factors that should be included in a benchmarking that is suitable for publication. (The consultation document is available from www.hefce.ac.uk/pubs/year/2015/201524/)

PPS 20th December 2015
I have now created a script in R that creates percentages close to those reported by BIS. The approach is, as I indicate above, still reliant on a statistical definition of 'below expectation' that means that, regardless of how well institutions are performing overall, there will always be some who perform in this range - unless everyone has 100% satisfaction ratings. Those who are interested in the technical details can find the relevant data and scripts on Open Science Framework: osf.io/aus52 

Open code: not just data and publications

I had a nice exchange on Twitter this week.

Nick Fay had found a tweet I had posted over a year ago, asking for advice on an arcane aspect of statistical analysis:

I'd had some replies, but they hadn’t really helped. In the end, I’d worked out that there was an error in the stats bible written by Rand Wilcox, which was leading me astray. Once I’d overcome that, I managed to get the analysis to work.

It was clear Nick was now having the same problem and going round in exactly the same circles I had experienced.

My initial thought was that I could probably dig out the analysis and reconstruct what I’d done, but my heart sank at the prospect. However, then I had a cheerful thought. I had deposited the analysis scripts for my project on the Open Science Framework, here. I checked, and the script was pretty well annotated, and as a bonus you got a script showing you how to make a nice swarm plot.

This experience comes hard on the heels of another interaction, this time around a paper I’m writing with Paul Thompson on p-curve analysis (latest preprint is here). Here there’s no raw data, just simulations, and it’s been refreshing to interact with reviewers who not only look at the code you have deposited, but also make their own code available.  There’ve been disagreements with the reviewers about aspects of our paper, and it helped enormously that we could examine one another’s code. The nice thing is that if code is available, you get to really understand what someone has done and also learn a great deal about coding.

These two examples illustrate the importance of making code open. It probably didn’t matter much when everyone was doing very simple and straightforward analyses. A t-test or correlation can easily be re-run from any package given a well-annotated dataset. But the trend in science is for analyses to get more and more complicated. I struggle to understand the methods of many current papers in neuroscience and genetics – fields where replication is sorely needed but impossible to achieve if everyone does things differently and only incompletely described. Even in less data-intensive areas such as psycholinguistics, there has been a culture change away from reliance on ANOVAs to much more fancy multilevel modelling approaches.

My experience leads me to recommend sharing of analysis code as well as data: it will help establish reproducibility of your findings, provide a training tool for others, and ensure your work is in a safe haven if you need to revisit it.

Finally, this is a further endorsement of Twitter as an academic tool. Without Twitter I wouldn't have discovered Open Science Framework, or PeerJ, both of which are great for those who want to embrace open science. And my interchange with Nick was not the end of the story. Others chipped in with helpful comments, as you can see below:

P.S. And here is another story of victory for Open Data, just yesterday, from the excellent Ed Yong.

Who's afraid of Open Data

Cartoon by John R. McKiernan, downloaded from: http://whyopenresearch.org/gallery.html

I was at a small conference last year, catching up on gossip over drinks, and somehow the topic moved on to journals, and the pros and cons of publishing in different outlets. I was doing my best to advocate for open access, and to challenge the obsession with journal impact factors. I was getting the usual stuff about how early-career scientists couldn't hope to have a career unless they had papers in Nature and Science, but then the conversation took an interesting turn.

"Anyhow," said eminent Professor X. "One of my postdocs had a really bad experience with a PLOS journal."

Everyone was agog. Nothing better at conference drinks than a new twist on the story of evil reviewer 3.  We waited for him to continue. But the problem was not with the reviewers.

"Yup. She published this paper in PLOS Biology, and of course she signed all their forms. She then gave a talk about the study, and there was this man in the audience, someone from a poky little university that nobody had ever heard of, who started challenging her conclusions. She debated with him, but then, when she gets back she has an email from him asking for her data."

We wait with bated breath for the next revelation.

"Well, she refused of course, but then this despicable person wrote to the journal, and they told her that she had to give it to him! It was in the papers she had signed."

Murmurs of sympathy from those gathered round. Except, of course, me. I just waited for the denouement. What had happened next, I asked.

"She had to give him the data. It was really terrible. I mean, she's just a young researcher starting out."

I was still waiting for the denouement. Except that there was no more. That was it! Being made to give your data to someone was a terrible thing. So, being me, I asked, why was that a problem? Several people looked at me as if I was crazy.

"Well, how would you like it if you had spent years of your life gathering data, data which you might want to analyse further, and some person you have never heard comes out of nowhere demanding to have it?"

"Well, they won't stop you analysing it," I said.

"But they may scoop you and find something interesting in it before you have a chance to publish it!"

I was reminded of all of this at a small meeting that we had in Oxford last week, following up on the publication of a report of a symposium I'd chaired on Reproducibility and Reliability of Biomedical Research. Thanks to funding by the St John's College Research Centre, a small group of us were able to get together to consider ways in which we could take forward some of the ideas in the report for enhancing reproducibility. We covered a number of topics, but the one I want to focus on here is data-sharing.

A move toward making data and analyses open is being promoted in a top-down fashion by several journals, and universities and publishers have been developing platforms to make this possible. But many scientists are resisting this process, and putting forward all kinds of argument against it. I think we have to take such concerns seriously: it is all too easy to mandate new actions for scientists to follow that have unintended consequences and just lead to time-wasting, bureaucracy or perverse incentives. But in this case I don't think the objections withstand scrutiny. Here are the main ones we identified at our meeting:

1.  Lack of time to curate data;  Data are only useful if they are understandable, and documenting a dataset adequately is a non-trivial task;

2.  Personal investment - sense of not wanting to give away data that had taken time and trouble to collect to other researchers who are perceived as freeloaders;

3. Concerns about being scooped before the analysis is complete;

4.  Fear of errors being found in the data;

5.  Ethical concerns about confidentiality of personal data, especially in the context of clinical research;

6.  Possibility that others with a different agenda may misuse the data, e.g. perform selective analysis that misrepresented the findings;

These have partial overlap with points raised by Giorgio Ascoli (2015) when describing NeuroMorpho.Org, an online data-sharing repository for digital reconstructions of neuronal morphology. Despite the great success of the repository, it is still the case that many people fail to respond to requests to share their data, and points 1 and 2 seemed the most common reasons.

As Ascoli noted, however, there are huge benefits to data-sharing, which outweigh the time costs. Shared data can be used for studies that go beyond the scope of the original work, with particular benefits arising when there is pooling of datasets. Some illustrative examples from the field of brain imaging were provided by Thomas Nichols at our meeting (slides here), where a range of initiatives is being developed to facilitate open data. Data-sharing is also beneficial for reproducibility: researchers will check data more carefully when it is to be shared, and even if nobody consults the data, the fact it is available gives confidence in the findings. Shared data can also be invaluable for hands-on training. A nice example comes from Nicole Janz, who teaches a replication workshop in social sciences in Cambridge, where students pick a recently published article in their field and try to obtain the data so they can replicate the analysis and results.

These are mostly benefits to the scientific community, but what about the 'freeloader' argument? Why should others benefit when you have done all the hard work? In fact, when we consider that scientists are usually receiving public money to make scientific discoveries, this line of argument does not appear morally defensible. But in any case, it is not true that the scientists who do the sharing have no benefits. For a start, they will see an increase in citations, as others use their data. And another point, often overlooked, is that uncurated data often become unusable by the original researcher, let alone other scientists, if it is not documented properly and stored on a safe digital site. Like many others, I've had the irritating experience of going back to some old data only to find I can't remember what some of the variable names refer to, or whether I should be focusing on the version called final, finalfinal, or ultimate. I've also had the experience of data being stored on a kind of floppy disk, or coded by a software package that had a brief flowering of life for around 5 years before disappearing completely.

Concerns about being scooped are frequently cited, but are seldom justified. Indeed, if we move to a situation where a dataset is a publication with its own identifier, then the original researcher will get credit every time someone else uses the dataset. And in general, having more than one person doing an analysis is an important safeguard, ensuring that results are truly replicable and not just a consequence of a particular analytic decision (see this article for an illustration of how re-analysis can change conclusions).

The 'fear of errors' argument is, of understandable but not defensible. The way to respond is to say of course there will be errors – there always are. We have to change our culture so that we do not regard it as a source of shame to publish data in which there are errors, but rather as an inevitability that is best dealt with by making the data public so the errors can be tracked down.

Ethical concerns about confidentiality of personal data are a different matter. In some cases, participants in a study have been given explicit reassurances that their data will not be shared: this was standard practice for many years before it was recognised that such blanket restrictions were unhelpful and typically went way beyond what most participants wanted – which was that their identifiable data would not be shared.  With training in sophisticated anonymization procedures, it is usually possible to create a dataset that can be shared safely without any risk to the privacy of personal information; researchers should be anticipating such usage and ensuring that participants are given the option to sign up to it.

Fears about misuse of data can be well-justified when researchers are working on controversial areas where they are subject to concerted attacks by groups with vested interests or ideological objections to their work. There are some instructive examples here and here. Nevertheless, my view is that such threats are best dealt with by making the data totally open. If this is done, any attempt to cherrypick or distort the results will be evident to any reputable scientist who scrutinises the data. This can take time and energy, but ultimately an unscientific attempt to discredit a scientist by alternative analysis will rebound on those who make it.  In that regard, science really is self-correcting. If the data are available, then different analyses may give different results, but a consensus of the competent should emerge in the long run, leaving the valid conclusions stronger than before.

I'd welcome comments from those who have started to use open data and to hear your experiences, good or bad.

P.S. As I was finalising this post, I came across some great tweets from the OpenCon meeting taking place right now in Brussels. Anyone seeking inspiration and guidance for moving to an open science model should follow the #opencon hashtag, which links to materials such as these: Slides from keynote by Erin McKiernan, and resources at http://whyopenresearch.org/about.html.