31 January 2022 – There is increasing interest across the research community about the use of randomisation, or lottery type mechanisms, in the allocation of research funds. A number of funders have begun to experiment with the use of randomisation alongside peer review in their decision-making processes. The preferred application is known as partial or focussed randomisation, where only a subset of applications is entered into the pool subject to the random draw. The Research on Research Institute (RoRI), an international consortium of research funders, academic institutions and technologists, has investigated the use of partial randomisation to learn more about how and why it is used, and what the pros and cons of its use are.
A number of RoRI’s partners have collaborated within the RoRi Randomisation project in order to bring together different insights into the use of this method. Outputs were presented in an online workshop on 1 December 2021. This workshop was jointly hosted by the Swiss National Science Foundation (SNSF), the European Molecular Biology Organization (EMBO) and RoRI. It featured presentations from researchers and funders to showcase case studies on the use of the method, an informational handbook presenting different experimental methods (including partial randomisation), and a small-scale qualitative study exploring funders’ organisational motivations for the use of partial randomisation.
A handbook to innovate
RoRI’s Director James Wilsdon opened the workshop and set the stage for Teo Firpo, from Nesta, and Sandra Bendiscioli, from EMBO, to share highlights from their work on “The experimental research funder’s handbook“. This handbook is in two parts, the first part explores ways in which funders could use experiments to develop their practice. Teo began by setting out how experiments can help funding organisations to explore alternatives to current approaches, test the impact of new activities, and improve processes. To set up an experiment in order to learn systematically about a change in practice, the broad phases would comprise: diagnosis of the issue, design of a solution and, finally testing of that solution. In the handbook, details of tools to assist in these three phases are presented, such as using interviews or shadowing someone to help diagnose a problem. Useful methods and ideas are also suggested for design such as prototyping your intervention and piloting it, as well as methods to evaluate whether it has worked. The various evaluation methods follow the taxonomy in Nesta’s Experimenter’s Inventory.
Sandra described how the second part of the handbook had been designed to focus on partial randomisation as one example of experimentation. This includes a summary of the benefits and limitations of partial randomisation as well as a checklist for those interested in setting up their own trials. Four detailed case studies in the use of partial randomisation were provided by SNSF, the Volkswagen Foundation, the Austrian Science Fund (FWF) and the Health Research Council of New Zealand (HRCNZ). These case studies describe details of the funding scheme, and context of how randomisation was applied and why it was introduced. The case studies also include a description of the randomisation procedure, which can be done using a variety of methods. Sandra summarised the highlights from the case studies and noted that overall they show that partial randomisation – after the usual rigorous peer review process has been followed – has been well accepted by applicants, peer reviewers, the public and the media. However, while partial randomisation has now been extended to other funding schemes, more evidence is needed to learn more about its potential to address issues of diversity and efficiency savings.
Motivated by fairness
Helen Buckley Woods, from RoRI, gave a presentation summarising the findings from a small-scale study (Why draw lots?) exploring the motivations and restraining factors behind the use of partial randomisation. The qualitative study collected data through interviews with practitioners and leaders from six funding organisations either planning or implementing partial randomisation. The key motivating factor identified from the data was fairness in decision-making when peer review had reached the limits of its use. It also ameliorated the biases that can creep in when panels are faced with projects that are indistinguishable and it is challenging to decide which projects to fund. The data also suggested that, as it is a more transparent process, it is perceived as fairer, and it is easier to communicate why a project was or was not funded. Legal considerations were also a driver to introduce partial randomisation and a prerequisite to its use, particularly for public funding bodies. Organisations that took part were at different stages in the application of partial randomisation. For those at a more developed stage, early reflections on its overall value, its effects on the funding community, and what it means for the role of peer review and the wider research system were beginning to emerge.
Testing the waters
In the second half of the workshop, chaired by Michele Garfinkel, from EMBO, different applications of partial randomisation were presented in four short presentations.
Firstly, Lucy Pomeroy, from the Health Research Council, in New Zealand, outlined their use of randomisation in their Explorer Grants scheme. Lucy reported that, in a recent survey, there had been an increase in the acceptability of the method’s use in this scheme amongst both applicants and assessors. As health funders, the HRC has the opportunity to make a significant contribution to health equity in support of the NZ government’s strategic health equity objectives. Two examples were given showing how randomisation could be used in conjunction with these strategic objectives.
Marco Bieri and Rachel Heyard, from the SNSF, then presented an update on the use of partial randomisation and what they had learned from the 2019 pilot on the Postdoc.Mobility scheme. In the pilot, the method was used in only a small set of applications (~4%). Clear communication was very important to prevent misunderstandings with stakeholders. There was a mixed reaction from panel members, but acceptance was growing with time. In addition, there were fewer reactions than expected from applicants and these were mainly about the process. The process of deciding which applications should be subject to the random selection is very important and Rachel Heyard explained the use of Bayesian ranking as one useful way of defining this group. These processes are now part of a new unified evaluation procedure at the SNSF.
The penultimate talk was given by Dagmar Simon, from Evaconsult, who gave an update on the evaluation of the Volkswagen Foundation’s use of partial randomisation in their Experiment! scheme. The aim of the scheme was to identify ground-breaking research ideas. In her research with colleague Martina Robbecke, interim results found that the method increased the diversity of the range of projects funded, and encouraged the submission of risk-taking proposals. In the most part it was welcomed by the research community. More details of the accompanying research project are available in the Volkswagen case study in “The experimental research funder’s handbook”.
Finally, Elisabeth Nindl, from the FWF, presented on the use of partial randomisation in their 1000 Ideas programme. The programme was introduced to promote radically new and risky research ideas. The selection entailed a double-blind peer review process with the random draw element conducted at an FWF board meeting using a specific procedure designed to ensure the trustworthiness of the method. Elisabeth reported that there were no complaints from applicants but some criticism that they had not received feedback on where their project was unsuccessful. Criticism from the FWF board and the jury was directed towards the objectives of the programme itself and on the quality of the applications, which were perhaps not as ‘risky’ as anticipated.
The session closed with comments and questions from the audience, and panel members highlighted the benefits of sharing ideas and practice at these kinds of events. The President of the National Research Council of the SNSF, Matthias Egger, also underscored the benefits of such exchanges, to share and develop possible methodologies to assess the longer-term effects of partial randomisation in multi-centre or multi-funder studies.
For more information about the project, including our work supporting partners to design experiments, please see the project page. If you have any questions or would like to join in future research in this area, please contact Helen Buckley Woods. Reposted from FEBS network with kind permission from FEBS and the author.
