Registered reports and preregistrations
A chat with Anne-Marike Schiffer from Nature Human Behaviour
by Cornelia van Scherpenberg, Ilona Lipp, Jacob Bellmund, Lieneke Jannsen and Mariella Paul
We had the honour of meeting Anne-Marike Schiffer, who is a senior editor at Nature Human Behaviour and regularly handles registered reports at the journal. This gave us the opportunity to ask some pending questions and get the perspective of somebody who sits on the other side of the journal submission systems.
Can you tell us a bit about registered reports in Nature Human Behaviour?
Our commitment to promote robust science is a key pillar of journal identity since the start. Registered reports and preregistrations fit right into that. That’s why we offered Registered Reports from the moment the journal was launched.
How do you as a journal make sure that reviewers know how to deal with registered reports?
When we send out articles for review, we send reviewers a set of questions that should help guide them through the review process. For registered reports, there is specific list of questions to answer. Reviewers do not always use these guidelines, especially more experienced reviewers sometimes do not. This is why our decision letter to authors includes instructions about which parts of the review are most important. In certain cases, we will overrule the reviewer’s opinion, providing appropriate justification both to the authors and reviewer. For example when the reviewer says “This is not novel and exciting.”, then we gently inform them that this is not the type of evaluation the journal is looking for in its assessment of Registered Reports and we tell the authors that they don't have to address this issue.
There has been a recent debate about whether researchers should get paid for reviewing articles. From editor’s perspective, would you expect the quality of reviews be higher if this was implemented?
I do understand that there are many different positions, and I understand why some reviewers feel that way. I also hear counter-arguments such that the reviewers’ own work also gets reviewed, so it is some sort of service to the community. Overall, I think that what is important is that reviewers get recognition for what they do. There are multiple developments across different journals and publishers, such as offering reviewers to have their name mentioned on the paper, or get the fact that they reviewed recorded in a database which is helpful especially for early career researchers, not least because it provides proof that reviewing activities listed on their CVs are accurate.
Timing is an argument often heard against going for registered reports. Are reviewers faster with registered reports to make the whole process more feasible?
In the beginning, the process was slow due to lack of reviewers with experience with the format of registered reports. This has become better. Now, there are usually not more than two rounds of review during stage 1 and just one during stage 2. I have heard this criticism a lot before, but on the other hand, because during stage 2 reviewers cannot ask for more analyses or experiments other than those strictly necessary to support the authors' interpretation, for example of unexpected results, the process towards the end is much faster. I am not sure what the numbers are, but I think Chris Chambers concluded that the review does not actually take longer.
Does it happen that during the first stage reviewers ask for things that are not covered by the ethics of the project? Getting ethics approval in case of changes to the protocol can be a very time-consuming process.
Ethical approval is required for peer review of the submitted protocol. Some institutions work with research ethic applications that do not have much experimental detail, especially for the type of study that I handle (cognitive neuroscience and psychology), or have waivers for simple behavioural studies. In this case, it's often not necessary to request a new Institutional Review Board approval. If the projects change beyond what is covered in the ethics approval, then yes, the amendments would need to be approved. However, I personally never had a Registered Report drop out of the review process because the project did not receive ethics approval for changes that resulted from peer- review.
Talking about ethics. Nowadays, there is also a trend for sharing of data. But a big issue of debate has been the sharing of MRI and genetics data. It is being argued that these types of data can never really be anonymized.
This is a complex issue, where conditions vary by discipline. Authors must share data for review purposes if requested to do so. We strongly encourage authors to publicly share their data upon publication, unless there are ethical or legal restrictions that prevent them from sharing. Structural brain data and genetic data are particularly challenging, as we now know that they cannot be fully de-identified. We follow current legal frameworks with respect to uniquely identifiable data that cannot be effectively de-identified. For Registered Reports, public sharing of all entirely de-identified data is mandatory. For all other manuscripts, at a minimum, the manuscript must include the statement: “The data that support the findings of this study are available from the corresponding author upon request”, mentioning any restrictions on availability. Even in fields where we find that data sharing is increasingly the norm, such as cognitive neuroscience, sometimes ethic approvals are a few years old and simply do not include anything about data sharing. This information should then be included in the Data availability statement. I expect to see this less and less, as the way people write their ethics applications is changing thanks to the open science movement and because more journals review data and code.
How do you deal with deviation from the protocol for Registered Reports?
For Registered Reports, all deviations from the protocol must be communicated, and in some cases we may ask reviewer what they think of the deviation. There may be valid, benign reasons for deviation from protocol, e.g. your Registered Report contains a description of how biological samples will be preprocessed based on the manufacturers specifications, and after Stage-1 in principle acceptance the manufacturer has switched to a new kit with a different preprocessing pipeline. But even such small deviations will have to be indicated in the method section and should ideally be communicated to the editor before the data is being collected.
What about preregistrations? Do you or the reviewers check whether the authors have actually done what they preregistered?
For clinical trials, there always has to be a clinical trials registration, and reviewers are specifically asked to check the agreement between submission and preregistration. In Psychology and Neuroscience if a preregistration is mentioned in the text, we ask for the link to be included and we check it's working and leads to a project that seems to be roughly the same. Reviewers are also asked to check the details of the preregistration. The reason why we're asking reviewers to vet the pre-registration is that these form part of the research methods, for which we always rely on experts.
When we get papers that have preregistered their analysis, we take that into account as a positive, because it increases everyone's trust in the analysis. If what the authors did differs from what they had preregistered, this needs to be justified and additional analyses have to be labelled as exploratory. It is our journal’s aim to push for transparency.
What do you think about preregistering after you have started acquiring data? Is it still worth the effort?
If you have all the data but haven’t looked at it, then it’s still better to preregister, not least to protect yourself from hindsight bias.
Also, a worry that we have heard is that you could preregister several times and then just pick the link that fits the end result?
If it’s for example on OFS, you cannot get rid of a preregistration. But yes, people would still need to find your other preregistrations and you can keep them under embargo for some time. Of course you can find a way to game every system. Personally, I think it’s a form of fraud. And being able to game a system does not mean the general endeavour is void. The fact that preregistrations exist is still much better than a world without preregistrations.
How long do you think it will take before preregistration be required?
Now, preregistration seems far more common in Psychology than in many other fields. I am not sure whether it will ever be entirely mandatory, and how long that might take. Serendipity is an important aspect of science, especially in discovery science and I wouldn't want to make a judgement of any type about fields I don't know, such as Physics or Molecular Chemistry. Thinking about Cognitive Neuroscience and Psychology, I would expect preregistration to be a wide-spread norm within a few years, even if not mandatory. I also believe in the importance of serendipitous findings in these fields, but I firmly believe they're strengthened by preregistered replications.
Another hot topic is open access. Currently, Nature Human Behaviour is not an open access journal. Do you think this will change? For example, boycotting of NeuroImage editors had some effect on policy, do you think this could happen in your journal as well. Do you sometimes hear from authors who decide against submitting to your journal because of this?
Of course I hear authors being critical of our model, but we also hear the reverse, from authors who do not work on huge grants and do not have the money to publish it open access the way it currently works. We still don’t have a system that works for everybody. However, Nature Human Behaviour, along with all other Nature Research journals (including Nature), will be switching to a transformative model of publication from January 1st, 2021. Ultimately, the aim is for the journal to become fully open access over a number of years.
When we think about papers, we used to think about actual papers, now we usually think about Pdfs. Recently, more interactive formats have started emerging, such as in Elife. Have you thought about more interactive forms as well?
I think this is great. For research, what we offer at the moment are standard articles, registered reports, and resources. These include Figures, Tables, and Extended Data. If authors submit additional material, such as an R markdown script, we will do our best to publish this with the paper. This was the case for a recent paper I handled.
Recently, you published a focus on publication pressure, aimed at graduate students, but really relevant to all of us. This got some attention, do you plan to follow up on it and can initiatives like ours contribute?
We actually published all these articles together exactly so that they would get a lot of attention. Currently, there is no plan to publish something exactly like this in the immediate future. I think it is part of our role as a journal to create a forum for such debates, but we are of course not policy makers. If you wanted to contribute to the publish or perish debate, the best place at the moment would be via the Nature communities pages.
This sounds great. Thank you for talking to us.