28 June 2018 – Conceived by a group of journal editors and academic publishers, and since signed by around 500 institutions and 12,000 individuals, the declaration calls for the wholesale reform of research assessment practices, particularly the egregious use of journal impact factors (JIFs) and similar metrics as a surrogate to judge an individual’s research output. Following on from DORA, the bibliometric community itself warned in the 2015 Leiden Manifesto of the dangers of using bibliometrics as a sole arbiter of quality. In the words of the economist Charles Goodhart: “when a measure becomes a target, it ceases to be a good measure.”
Bernd Pulverer, EMBO Head of Scientific Publications says: “To put it bluntly: by making the decision to accept or reject a paper, we as editors, are playing a sometimes decisive but unwanted role in the judgment of people’s careers. This is problematic as we select for editorial not hiring criteria.” Pulverer thinks that DORA has provided the impetus for the research community, together with its institutions and funders, to begin to break the addiction to JIFs. Power lies where the money is, and so he is particularly happy that research funders are taking steps to kick the JIF habit.
At EMBO, applicants for Long-Term Fellowships or the Young Investigator Programme are asked to provide summaries of their most important research outcomes without reference to the impact factor of their publications. And EMBO Council’s deliberation of the Gold Medal recipient now begins with four-minute summaries of each of the nominee’s scientific merits in terms of the originality of their work and the impact it has had on their fields, explicitly and deliberately banning any discussions on where they have published; a practice that has wholly changed the subsequent discussions.
Funders on the case
In the UK, the Wellcome Trust, Cancer Research UK, and all seven of the UK research councils have signed up to DORA, instructing reviewers, panel members, recruitment and promotion panels that they should be assessing the content of specific papers, rather than where they’re published, when considering the impact of an individual researcher’s contribution. And both Cancer Research UK and Wellcome now ask applicants to select their top research outputs and highlight why they are interesting, rather than list all their papers. Such outputs can include datasets, patents and contributions to changing medical practice or health policy.
Fiona Reddington, Head of Population Research Funding at Cancer Research UK, says that so far, researchers have been enthusiastic, but that culture change will take some time – committee members still fall into old habits of describing research by JIF, and applicants “aren’t yet quite sure what good looks like when filling out the new-style application forms”.
Jim Smith, Director of Science at Wellcome, agrees: “We’ve been doing it the old way for decades, and it’s hard for people to change, even when they want to, which I think they do.” Smith and his team have also been thinking how they can influence others to adopt DORA’s principles. “Wellcome does have the ability to make grand gestures,” he says, “so we could, for example, stipulate that the host institutions of our grantholders are DORA signatories.”
Hiring practices revisited
Universities are also starting to rethink how they hire and reward their researchers without focusing on the JIF. In the USA, the Department of Cell Biology at University of Texas Southwestern Medical Center in Dallas has revamped its recruitment procedures for junior faculty.
Candidates write a brief cover letter outlining their past contributions and future vision, which can be viewed by all the faculty. They have a short video call interview if even just one member of the faculty expresses interest, and shortlisted candidates then proceed to a full interview. Doing away with a standard publication-based CV as a triage method has led to a better, more diverse field of candidates being interviewed, and the hiring of motivated, enthusiastic scientists of great potential [see doi: 10.1091/mbc.E17-08-0534].
In the Netherlands, University Medical Center (UMC) Utrecht is leading the way in redesigning its internal assessment and promotion methods [see doi: 10.1038/538453a]. Biomedical research in the Netherlands was particularly in thrall to JIFs, which was hampering the careers of some excellent researchers. Those who spent time on work such as making health policy, writing official reports rather than papers, or establishing links with industry, were not always appreciated for their ‘non-scientific’ activities and were failing to win the promotions they deserved.
Candidate (associate) professors at UMC Utrecht are now assessed using a portfolio that better reflects the multifaceted nature of a scientific career. Research programmes are also evaluated based on their wider clinical and social impact, rather than a simplistic bibliometric approach.
Rinze Benedictus, a staff advisor at UMC Utrecht, says that there has been an overall increase in the number of professors appointed, with more coming from otherwise meritorious fields that do not traditionally publish in high impact journals. Benedictus hopes that this will lead to increasing diversity in senior positions, completing the virtuous circle. However, he says, “you have to be aware of what is going on in the rest of the science system – you can be a front runner but you can’t be too far ahead.”
Researchers’ careers, especially early in their careers, are inherently mobile, so a new post-JIF CV has to be flexible enough to still work in less progressive places. And Benedictus echoes the concerns of many when he points out that faculty are still faced with the conundrum of whether to boost their lab members’ careers by publishing in high impact publications, thereby reinforcing the existing system.
Making change happen
Stephen Curry of Imperial College, who chairs the DORA steering committee, is mindful that signing DORA is not enough [also see doi: 10.1038/d41586-018-01642-w]. “Some people have criticized DORA as being ‘clicktivism’,” he says, “so we need to walk the walk as well as talk the talk.” The DORA website will become a forum highlighting good practice, and hubs will be established around the world to start to reach into countries, such as India and China, that are still heavily reliant on bibliometrics.
What of the next five years? “I would hope many more institutions will change how they go about research assessment,” Curry suggests, but he also has a more immediate wish: “At present, we’re sending a terrible message to the next generation about what we reward in research,” he says, “and that needs to change as fast as possible.”
