Advertisement
Canadian Journal of Cardiology

Challenges and Lessons Learned From COVID-19 Trials: Should We Be Doing Clinical Trials Differently?

  • Perrine Janiaud
    Affiliations
    Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland
    Search for articles by this author
  • Lars G. Hemkens
    Affiliations
    Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland

    Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, USA

    Meta-Research Innovation Center Berlin (METRIC-B), Berlin Institute of Health, Berlin, Germany
    Search for articles by this author
  • John P.A. Ioannidis
    Correspondence
    Corresponding author: Dr John P.A. Ioannidis, Departments of Medicine, Health Research and Policy, Biomedical Data Science, and Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University, 1265 Welch Rd, Medical School Office Building Room X306, Stanford, California 94305, USA.
    Affiliations
    Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, USA

    Meta-Research Innovation Center Berlin (METRIC-B), Berlin Institute of Health, Berlin, Germany

    Department of Medicine, Stanford University School of Medicine, Stanford, California, USA

    Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California, USA

    Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California, USA

    Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, California, USA
    Search for articles by this author

      Abstract

      The COVID-19 crisis led to a flurry of clinical trials activity. The COVID-evidence database shows 2814 COVID-19 randomized trials registered as of February 16, 2021. Most were small (only 18% have a planned sample size > 500) and the rare completed ones have not provided published results promptly (only 283 trial publications as of February 2021). Small randomized trials and observational, nonrandomized analyses have not had a successful track record and have generated misleading expectations. Different large trials on the same intervention have generally been far more efficient in producing timely and consistent evidence. The rapid generation of evidence and accelerated dissemination of results have led to new challenges for systematic reviews and meta-analyses (eg, rapid, living, and scoping reviews). Pressure to regulatory agencies has also mounted with massive emergency authorizations, but some of them have had to be revoked. Pandemic circumstances have disrupted the way trials are conducted; therefore, new methods have been developed and adopted more widely to facilitate recruitment, consent, and overall trial conduct. On the basis of the COVID-19 experience and its challenges, planning of several large, efficient trials, and wider use of adaptive designs might change the future of clinical research. Pragmatism, integration in clinical care, efficient administration, promotion of collaborative structures, and enhanced integration of existing data and facilities might be several of the legacies of COVID-19 on future randomized trials.

      Résumé

      La crise de la COVID-19 a entraîné une vague d'essais cliniques. La base de données COVID-evidence répertorie 2 814 essais randomisés sur la COVID-19 consignés en date du 16 février 2021. Il s'agit pour la plupart d'essais de petite envergure (la taille prévue de l’échantillon dépassait 500 pour seulement 18 % d'entre eux) et les rares essais achevés n'ont pas rapidement mené à la publication de résultats (seulement 283 essais publiés en février 2021). Les petits essais randomisés et les analyses d'observation non randomisées n'ont pas fait leurs preuves et ont généré des attentes trompeuses. Les divers essais de plus grande envergure sur la même intervention ont généralement été beaucoup plus efficaces pour produire des données probantes constantes dans les meilleurs délais. La production rapide de données probantes et la diffusion accélérée des résultats ont entraîné de nouveaux défis pour les revues systématiques et les méta-analyses (p. ex., revues rapides, revues avec mises à jour et revues de la portée). La pression sur les organismes de réglementation s'est également accentuée en raison de très nombreuses autorisations d'urgence, mais certaines d'entre elles ont dû être révoquées. Les circonstances de la pandémie ont bouleversé la façon dont les essais sont menés; par conséquent, de nouvelles méthodes ont été élaborées, puis adoptées plus largement afin de faciliter le recrutement et le consentement, ainsi que la conduite générale des essais. Selon l'expérience apportée par la COVID-19 et ses défis, la planification de plusieurs essais efficients de grande envergure ainsi qu'une utilisation plus large des plans adaptatifs pourraient changer l'avenir de la recherche clinique. Le pragmatisme, l'intégration aux soins cliniques, l'administration efficiente, la promotion des structures de collaboration et une plus grande intégration des données et des installations existantes pourraient compter parmi les répercussions de la COVID-19 sur les futurs essais randomisés.
      The COVID-19 pandemic has required an ultra-urgent response to generate evidence to handle a major public health crisis. Effective treatments, vaccines, and other measures were important to develop, test, and implement as quickly as possible. An unprecedent number of randomized controlled trials (RCTs) to assess therapeutics for COVID-19 were initiated, and to a lesser extent for preventive measures, in particular, vaccines.
      • Janiaud P
      • Axfors C
      • Van't Hooft J
      • et al.
      The worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days.
      ,
      • Janiaud P
      • Axfors C
      • Ioannidis JPA
      • Hemkens LG
      Recruitment and results reporting of COVID-19 randomized clinical trials registered in the first 100 days of the pandemic.
      For many other influential decisions (eg, nonpharmaceutical interventions), evidence sadly depended almost exclusively on precarious modelling and observational data.
      • Cristea IA
      • Naudet F
      • Ioannidis JPA
      Preserving equipoise and performing randomised trials for COVID-19 social distancing interventions.
      In this report, we review the challenges of and what we have learned from the COVID-19 clinical research agenda in the first 14 months of the pandemic and how we can put this knowledge to good use moving forward. The lessons learned are potentially important not only for COVID-19, but, perhaps most importantly for the future of RCTs in general.

      The COVID-19 Agenda of Randomized Trials

      In March 2020, the COVID-evidence database (www.covid-evidence.org) was launched with the aim to gain insight on the COVID-19 clinical research agenda.

      Janiaud P, Axfors C, Saccilotto R, Hemkens L. COVID-evidence: a living database of trials on interventions for COVID-19. Available at: https://osf.io/gehfx. Accessed March 1, 2021.

      Using a multimethod approach combining peer-reviewed search strategies of study registries and publication databases, continuous automated extraction of search results, automated classifications combined with manual screening and data extraction, and quality control through expert review, it provides information about worldwide planned, ongoing, and completed RCTs on any intervention to treat or prevent SARS-CoV-2-infection. In the first 100 days of the pandemic, more than 500 RCTs had already been registered on ClinicalTrials.gov and the World Health Organization International Clinical Registry Platform. Most trials were to assess interventions for treating patients with COVID-19; conversely only 11% focused on interventions to prevent COVID-19 infection, and none assessed social distancing or lockdown measures. RCTs were mostly small and often investigated the same interventions.
      • Janiaud P
      • Axfors C
      • Van't Hooft J
      • et al.
      The worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days.
      Similar observations were made in other reviews of early registered COVID-19 RCTs.
      • Song G
      • Cheng MQ
      • Wei XW
      Analysis of the WHO ICTRP for novel coronavirus clinical trial registrations.
      • Maguire BJ
      • McLean ARD
      • Rashan S
      • et al.
      Baseline results of a living systematic review for COVID-19 clinical trial registrations.
      • Karlsen APH
      • Wiberg S
      • Laigaard J
      • et al.
      A systematic review of trial registry entries for randomized clinical trials investigating COVID-19 medical prevention and treatment.
      • Batta A
      • Khirasaria R
      • Kapoor V
      • Varshney D
      Therapeutic clinical trials to combat COVID-19 pandemic in India: analysis from trial registry.
      • Wang Y
      • Zhou Q
      • Xu M
      • Kang J
      • Chen Y
      Characteristics of clinical trials relating to COVID-19 registered at ClinicalTrials.gov.
      • Lythgoe MP
      • Middleton P
      Ongoing clinical trials for the management of the COVID-19 pandemic.
      Although these clinical research efforts were much needed, the excessive duplication of efforts and lack of collaboration in the early COVID-19 clinical research agenda was putting it at risk of creating research waste
      • Glasziou PP
      • Sanders S
      • Hoffmann T
      Waste in COVID-19 research.
      characterized by unnecessary duplication of trials, poor study designs, and insufficient reporting of results.
      The cumulative number of RCTs registered has steadily increased (Fig. 1A), with a total of 2814 RCTs included in COVID-evidence as of February 16, 2021. Of note, the monthly number of RCTs registered has been slowly decreasing after a peak in the number of registrations in April 2020 (n = 592; Fig. 1A). Although the clinical research agenda was initially largely dominated by trials conducted in Asia (mostly China), by March 2020, the number of trials registered in the rest of the world quickly increased after the spread of the pandemic (Fig. 1B). Most trials have continued to be small (Fig. 2) with only 18% of the RCTs planned to include > 500 participants. Treatment with hydroxychloroquine was explored by a total of 304 RCTs of 2814 RCTs (in the first 100 days, every 1 in 6 RCTs investigated this intervention). The results from the Randomised Evaluation of COVID-19 Therapy (RECOVERY) showed no clinical benefit
      • Collaborative Group RECOVERY
      • Horby P
      • Mafham M
      • et al.
      Effect of hydroxychloroquine in hospitalized patients with COVID-19.
      and a meta-analysis showed an association of use of hydroxychloroquine with increased mortality.
      • Axfors C
      • Schmitt AM
      • Janiaud P
      • et al.
      Mortality outcomes with hydroxychloroquine and chloroquine in COVID-19 from an international collaborative meta-analysis of randomized trials.
      However, an additional 60 RCTs were registered after the RECOVERY trial press release.
      RECOVERY
      Statement from the Chief Investigators of the Randomised Evaluation of COVid-19 thERapY (RECOVERY) Trial on hydroxychloroquine.
      Overall, as of March 2, 2021, there were 6 drug treatments (remdesivir, COVID-19 convalescent plasma, bamlanivimab, baricitinib in combination with remdesivir, casirivimab and imdevimab, and bamlanivimab and etesevimab) with Emergency Use Authorizations (EUAs) by the US Food and Drug Administration (FDA)

      FDA. US Food & Drug Administration. Emergency Use Authorization. Available at:https://www.fda.gov/emergency-preparedness-and-response/mcm-legal-regulatory-and-policy-framework/emergency-use-authorization. Accessed February 26, 2021.

      and only 1 (remdesivir) with a conditional marketing authorization by the European Medicines Agency (EMA).

      European Medicines Agency. Treatments and vaccines for COVID-19. Available at: https://www.ema.europa.eu/en/human-regulatory/overview/public-health-threats/coronavirus-disease-covid-19/treatments-vaccines-covid-19. Accessed February 26, 2021.

      Figure 1
      Figure 1Number of registrations over time. (A) Cumulative registration (blue) and number of registrations per month (red). (B) Cumulative number of registrations per month according to continent. Sixteen trials were registered before January 1, 2020 and adapted their protocol to include patients with COVID-19.
      Figure 2
      Figure 2Planned sample size for the 2814 registered randomized controlled trials (RCTs; trials registered as of February 16, 2021). For 97 registered RCTs the planned sample size was not available. The dotted lines represent the intersection of the cumulative frequency of 82% of the RCTs that included < 500 participants.
      Drug development is a long and costly endeavour, antinomic to the urgent need of finding therapeutics for an acute pandemic. Therefore, most of the RCTs focused on testing the repurposing of already existing drugs that had been already approved or had been under investigation for other indications (Table 1).
      • Singh TU
      • Parida S
      • Lingaraju MC
      • et al.
      Drug repurposing approach to fight COVID-19.
      This was on the basis of the assumptions that there is substantial evidence on the efficacy and safety of these drugs.
      • Rodgers F
      • Pepperrell T
      • Keestra S
      • Pilkington V
      Missing clinical trial data: the evidence gap in primary data for potential COVID-19 drugs.
      However, a review that focused on the reporting of clinical results for 19 potential COVID-19 drugs showed that 40% of the completed trials that assessed those drugs before the pandemic, did not report their results on ClinicalTrials.gov or in the scientific literature.
      • Rodgers F
      • Pepperrell T
      • Keestra S
      • Pilkington V
      Missing clinical trial data: the evidence gap in primary data for potential COVID-19 drugs.
      For hydroxychloroquine, 37% of the trials were unreported.
      Table 1Overview of RCTs assessing vaccines and well-known interventions (most explored in RECOVERY; on the basis of 2814 RCTs registered as of February 16, 2021)
      InterventionTrials, n (%)Planned cumulative sample sizePlanned median sample size [IQR]RECOVERY results release dateTrials registered before/after RECOVERY, nRECOVERY results: 28-day mortality, relative risk (95% CI)
      Hydroxychloroquine304 (10.8)436,403160 [63-510]June 5, 2020243/601.09 (0.97-1.23); 4716 patients
      Ritonavir117 (4.2)409,220100 [60-326]June 29, 202092/251.03 (0.91-1.17); 5040 patients
      Lopinavir100 (3.6)407,846115 [60-484]June 29, 202081/191.03 (0.91-1.17); 5040 patients
      Azithromycin86 (3.1)70,872160 [60-310]December 14, 202084/20.97 (0.87-1.07); 7763 patients
      Convalescent plasma82 (2.9)23,844128 [58.5-376]January 15, 202179/31.04 (0.95-1.14); 10,406
      Tocilizumab54 (1.9)77,805188 [78-310]February 11, 202154/00.86 (0.77-0.96); 4116 patients
      Dexamethasone35 (1.2)10,033121 [62-300]June 16, 202010/250.83 (0.75-0.93); 6425 patients
      Vaccine175 (6.2)899,808900 [200-3000]
      Chloroquine66 (2.3)196,971120 [60-300]
      Remdesivir44 (1.6)354,375650 [152-2160]
      Missing planned sample size for 25 RCTs for hydroxychloroquine; 5 for vaccine; 2 for ritonavir; 2 for lopinavir; 7 for azithromycin; 1 for tocilizumab; 1 for dexamethasone; and 4 for chloroquine. Of note, lopinavir and ritonavir were either studied together or separately. Chloroquine and remdesivir were not assessed in RECOVERY.
      CI, confidence interval; IQR, interquartile range; RCT, randomized controlled trial; RECOVERY, Randomised Evaluation of COVID-19 Therapy.
      Since the beginning of the pandemic, the most pressing clinical question has been, “How do we prevent deaths?” However, it has been previously shown that only 15.8% of registered RCTs that assessed a treatment intended to use mortality as a primary end point.
      • Janiaud P
      • Axfors C
      • Van't Hooft J
      • et al.
      The worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days.
      A commonly reported end point identified is the use of ordinal scales
      • Desai A
      • Gyawali B
      End points used in phase III randomized controlled trials of treatment options for COVID-19.
      ; the most common one being a 7-point ordinal scale (1, death; 2, hospitalized with invasive mechanical ventilation or extracorporeal membrane oxygenation; 3, hospitalized, with noninvasive ventilation or high-flow oxygen; 4, hospitalized and requiring low-flow supplemental oxygen; 5, hospitalized not requiring supplemental oxygen but requiring ongoing medical care (COVID-19-related or otherwise); 6, hospitalized not requiring supplemental oxygen no longer requiring ongoing medical care; and 7, not hospitalized). Although ordinal scales have the advantages to capture data on multiple clinical states ranging from cure to death, and to gain in statistical power,
      • Dodd LE
      • Follmann D
      • Wang J
      • et al.
      End points for randomized controlled clinical trials for COVID-19 treatments.
      there is a substantial lack of harmonization in the usage of ordinal scales in COVID-19 trials with heterogenous point systems and definitions.
      • Zarin DA
      • Rosenfeld S
      Lack of harmonization of coronavirus disease ordinal scales.
      Several initiatives have been implemented to develop core outcome sets but their effects have yet to be assessed.
      COMET Initiative
      Core outcome set developers’ response to COVID-19.
      Another predominant issue is the representativeness of the patients included in the COVID-19 clinical research agenda. An early review that assessed 12 published RCTs showed that the median age of the included patients was 56.3 years and most exclusions were on the basis of comorbidities significantly more prevalent in patients 65 years old and older, highlighting the under-representation of elderly individuals
      • Prendki V
      • Tau N
      • Avni T
      • et al.
      A systematic review assessing the under-representation of elderly adults in COVID-19 trials.
      although they are 30-100 times more likely to die.
      • Ioannidis JPA
      • Axfors C
      • Contopoulos-Ioannidis DG
      Population-level COVID-19 mortality risk for non-elderly individuals overall and for non-elderly individuals without underlying diseases in pandemic epicenters.
      Of note, the extraordinary role of RECOVERY is reflected also by its representativeness with approximately 20% of the patients aged 80 years and older.
      • Collaborative Group RECOVERY
      • Horby P
      • Mafham M
      • et al.
      Effect of hydroxychloroquine in hospitalized patients with COVID-19.
      ,
      • Horby PW
      • Mafham M
      • Bell JL
      • et al.
      Lopinavir–ritonavir in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial.
      ,
      The RECOVERY Collaborative Group
      Dexamethasone in hospitalized patients with Covid-19—preliminary report.
      Elderly people have always been less likely to be included in RCTs of diverse conditions
      • Beers E
      • Moerkerken DC
      • Leufkens HGM
      • Egberts TCG
      • Jansen PAF
      Participation of older people in preauthorization trials of recently approved medicines.
      ,
      • Ruiter R
      • Burggraaf J
      • Rissmann R
      Under-representation of elderly in clinical trials: an analysis of the initial approval documents in the Food and Drug Administration database.
      and it is unlikely to be different in the COVID-19 era. The issue is more severe for COVID-19 trials and the under-representation might be worse for elderly people who are also frail. Frail nursing home residents accounted for a large proportion of deaths in most European countries and the United States,
      • Ioannidis JPA
      • Axfors C
      • Contopoulos-Ioannidis DG
      Second versus first wave of COVID-19 deaths: shifts in age distribution and in nursing home fatalities.
      but were not represented properly in either therapeutic or vaccine trials. Other reviews have highlighted the lack of inclusion of pregnant and lactating women, and children.
      • Chokkara S
      • Volerman A
      • Ramesh S
      • Laiteerapong N
      Examining the inclusivity of US trials of COVID-19 treatment.
      In addition, the fatality rate is higher in men compared with women,
      • Bhopal SS
      • Bhopal R
      Sex differential in COVID-19 mortality varies markedly by age.
      yet early publications did not consistently report results stratified according to sex.
      • Schiffer VMMM
      • Janssen EBNJ
      • van Bussel BCT
      • et al.
      The “sex gap” in COVID-19 trials: a scoping review.
      Trials assessing drugs for treating COVID-19 have clearly dominated the clinical research agenda since the beginning of the pandemic.
      • Janiaud P
      • Axfors C
      • Van't Hooft J
      • et al.
      The worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days.
      ,
      • Song G
      • Cheng MQ
      • Wei XW
      Analysis of the WHO ICTRP for novel coronavirus clinical trial registrations.
      • Maguire BJ
      • McLean ARD
      • Rashan S
      • et al.
      Baseline results of a living systematic review for COVID-19 clinical trial registrations.
      • Karlsen APH
      • Wiberg S
      • Laigaard J
      • et al.
      A systematic review of trial registry entries for randomized clinical trials investigating COVID-19 medical prevention and treatment.
      ,
      • Wang Y
      • Zhou Q
      • Xu M
      • Kang J
      • Chen Y
      Characteristics of clinical trials relating to COVID-19 registered at ClinicalTrials.gov.
      ,
      • Lythgoe MP
      • Middleton P
      Ongoing clinical trials for the management of the COVID-19 pandemic.
      Conversely, very few to no data exist regarding nonpharmaceutical measures to help prevent COVID-19, such as for example, the use of masks
      • Bundgaard H
      • Bundgaard JS
      • Raaschou-Pedersen DET
      • et al.
      Effectiveness of adding a mask recommendation to other public health measures to prevent SARS-CoV-2 infection in Danish mask wearers: a randomized controlled trial.
      or school closures.
      • Fretheim A
      • Flatø M
      • Steens A
      • et al.
      COVID-19: we need randomised trials of school closures.
      The search for a vaccine was another cornerstone of the strategy against COVID-19. As of mid-February 2021, there were 175 vaccine RCTs registered planning to include 899,808 participants (Table 1). Vaccine trials represented 6.2% of the clinical research agenda compared with 10.8% for hydroxychloroquine trials but included twice as many participants (899,808 vs 436,403 participants, respectively). Nineteen of the registered RCTs had a publication. Despite the unprecedented speed of the clinical research agenda, it resulted in only very few authorized vaccines: Pfizer-BioNTech, Janssen, and Moderna vaccines obtained EUAs by the FDA

      FDA. US Food & Drug Administration. Emergency Use Authorization. Available at:https://www.fda.gov/emergency-preparedness-and-response/mcm-legal-regulatory-and-policy-framework/emergency-use-authorization. Accessed February 26, 2021.

      and by the EMA

      European Medicines Agency. Treatments and vaccines for COVID-19. Available at: https://www.ema.europa.eu/en/human-regulatory/overview/public-health-threats/coronavirus-disease-covid-19/treatments-vaccines-covid-19. Accessed February 26, 2021.

      whereas the AstraZeneca vaccine was authorized only by the EMA.

      Fate of the Clinical Research Agenda

      Poorly designed and reported studies have always been symptomatic of research waste
      • Glasziou P
      • Altman DG
      • Bossuyt P
      • et al.
      Reducing waste from incomplete or unusable reports of biomedical research.
      but with the duplication of many small studies during the pandemic the phenomenon has been aggravated.
      On the basis of the information in clinical trial registries, as of mid-February 2021, 168 of 2814 registered RCTs (6%) were listed as being completed and 41 (1%) as terminated early. Of note, only 14 RCTs had results posted in the registry. Delays in the research agenda have been observed, affecting its potential to deliver much needed evidence.
      • Janiaud P
      • Axfors C
      • Ioannidis JPA
      • Hemkens LG
      Recruitment and results reporting of COVID-19 randomized clinical trials registered in the first 100 days of the pandemic.
      Because registries might be not consistently updated, it is important to acknowledge that some more trials have been completed.
      The COVID-evidence database also contains 283 publications of RCTs reporting results; 171 of which were linked to a registered RCT. In-depth evaluation was conducted for 50% of them (141/283), randomly selected: 73% (103/141) seemed to have completed the planned protocol, 19% (27/141) were terminated early, and 8% (11/141) were ongoing and presenting preliminary or arm-specific results. Of the 27 trials terminated early, the most common reasons, among others, were enrollment difficulties with a decrease in number of cases (n = 13), futility (n = 5), and emergence of new evidence (n = 5).

      Major Lessons Learned

      The COVID-19 experience has informed some of the longstanding debates about the optimal design and conduct of an RCT agenda. The key lessons learned are summarized in Box 1.
      Key lessons learned
      Tabled 1
      • Observational studies, although essential when pertinent, have generated misleading results later contradicted by subsequent randomized evidence.
      • Large trials can rapidly generate evidence on outcomes that matter.
      • Adaptive platform trials embedded in routine clinical care have efficiently and largely contributed to evidence generation and created synergies for collaborations.
      • Uncertainties and lack of harmonization in decision-making and dissemination of research results might contribute to skepticism.
      • The lockdown and social distancing measures have accelerated the implementation of innovative and remote solutions to conduct clinical trials.
      • Regulatory agencies and ethics and monitoring committees have tried to streamline their processes.

      Small vs large trials

      Traditionally, most medical decisions in the past have been informed from results of mostly small trials. Large trials have been uncommon. Although large trials were deemed desirable, and calls for large, simple trials were made decades ago,
      • Yusuf S
      • Collins R
      • Peto R
      Why do we need some large, simple randomized trials?.
      their conduct has been increasingly perceived to be difficult, requiring a long time to recruit and achieve sufficient follow-up. The choice of small trials has also generated high dependence on surrogate and composite outcomes rather than on simple, clinical end points that truly matter. During the COVID-19 pandemic, however, our cumulative experience shows that the paradigm of large, definitive trials can work and can provide rapid evidence on outcomes that matter.
      Conversely, it is questionable whether much was learned from the many hundreds of small trials that were initiated. Most of these small trials were underpowered to generate any results in a time frame that would be deemed useful for dealing with the pandemic, and many might never generate any worthwhile results or might even be abandoned because of futility. Small trials seem to have disadvantages at multiple fronts, including those previously perceived as potential advantages. Not only is their outcome selection and pragmatism questioned, but also the timing for accumulating robust evidence from multiple small trials can be actually slower than the timing of a large trial. Small trials and their meta-analyses have led to misleading inferences in the past in fields that had the benefit of conducting also large trials (eg, cardiovascular medicine). Interventions like magnesium in patients with myocardial infarction offer cautionary tales.
      ISIS-4: a randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction. ISIS-4 (Fourth International Study of Infarct Survival) Collaborative Group.
      The COVID-19 experience reinforced the need for timely, large trials.

      Observational data vs randomized trials

      In the past decade, there has been a major push to promote the use of observational data for obtaining large-scale, so-called real-world evidence in a timely fashion for comparative effectiveness questions. This trend has been questioned as to its validity by some methodologists and some empirical data,
      • Hemkens LG
      • Contopoulos-Ioannidis DG
      • Ioannidis JPA
      Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey.
      but has been heavily endorsed by many proponents. The track record of nonrandomized clinical trials and of observational data set analyses of effectiveness during the pandemic has been rather disappointing with regard to providing reliable evidence. This is reminiscent of past disappointments with high-profile observational data refuted by mega-trials in cardiovascular medicine and other fields (eg, hormone replacement or vitamin E).
      • Ioannidis JPA
      Contradicted and initially stronger effects in highly cited clinical research.
      For example, in August 2020, the FDA issued an EUA for convalescent plasma for the treatment of hospitalized patients with COVID-19 in a situation without sound randomized trial evidence
      • Mahase E
      Covid-19: US approves emergency use of convalescent plasma despite warnings over lack of evidence.
      but with observational data from > 35,000 patients published as a preprint.
      • Joyner MJ
      • Senefeld JW
      • Klassen SA
      • et al.
      Effect of convalescent plasma on mortality among hospitalized patients with COVID-19: initial three-month experience.
      Interestingly, within 5 months, large-scale randomized evidence contradicted not only the beneficial results of this analysis but also many of the perceived limitations made in the preprint against conducting RCTs in the pandemic situation to assess plasma. The large RECOVERY trial published results for the convalescent plasma arm in January 2021, showing the clear feasibility to randomize 10,406 patients and showing no benefit on mortality.

      RECOVERY. Convalescent Plasma results—RECOVERY Trial. Available at: https://www.recoverytrial.net/results/convalescent-plasma. Accessed February 26, 2021.

      Similarly for hydroxychloroquine, numerous very large observational studies contributed to high expectations but were contradicted by subsequent randomized evidence. The lowest point was the rapid publication and retraction of data from a large-scale observational analysis

      Mehra MR, Desai SS, Ruschitzka F, Patel AN. RETRACTED: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis [e-pub ahead of print] [erratum in: Lancet https://doi:org/10.1016/S0140-6736(20)31249-6] [retraction in: Lancet https://doi:org/10.1016/S0140-6736(20)31324-6]. Lancet. doi:10.1016/S0140-6736(20)31180-6, Accessed February 26, 2021.

      that apparently was entirely fake.

      Retraction Watch. Marcus A. Lancet, NEJM retract controversial COVID-19 studies based on Surgisphere data. Available at: https://retractionwatch.com/2020/06/04/lancet-retracts-controversial-hydroxychloroquine-study. Accessed February 26, 2021.

      Faking large-scale observational data is “easier” than faking an equivalent-size RCT, as illustrated by the former example and the fact that large-scale clinical trials have typically much stronger oversight, by regulators and independent monitoring committees.
      Of note, some very small randomized trials also suggested potential benefits (eg, for convalescent plasma). It is unknown whether this might reflect chance findings, especially with surrogate outcomes and superimposed selective reporting or publication. Sadly, there is some evidence from the pre-COVID-19 era that a large share of small RCTs, perhaps most, from some jurisdictions might be “zombie trials” because their data suggest fraud and/or are implausible.
      • Carlisle JB
      False individual patient data and zombie randomised controlled trials submitted to Anaesthesia.
      ,
      • Ioannidis JPA
      Hundreds of thousands of zombie randomised trials circulate among us.
      With an increasing number of RCTs done in jurisdictions without strong clinical research tradition and with weak oversight, these problems might become more prominent.
      However, nonrandomized evidence is likely to continue to be extremely important for mapping adverse events for licensed interventions when these are applied widely in the community, outside of a trial setting, and when long-term follow-up is afforded. This is particularly relevant for urgently authorized vaccines. A challenge with emerging observational data on harms is that the evidence base is often fluid, changing, and open to different interpretation by regulatory agencies, as in the case of adverse events from the Astra-Zeneca

      European Medicines Agency. EMA's safety committee continues investigation of COVID-19 Vaccine AstraZeneca and thromboembolic events – further update. Available at: https://www.ema.europa.eu/en/news/emas-safety-committee-continues-investigation-covid-19-vaccine-astrazeneca-thromboembolic-events. Accessed February 26, 2021.

      and Janssen vaccines.

      US Food & Drug Administration. Joint CDC and FDA Statement on Johnson & Johnson COVID-19 Vaccine. Available at: https://www.fda.gov/news-events/press-announcements/joint-cdc-and-fda-statement-johnson-johnson-covid-19-vaccine. Accessed February 26, 2021.

      Uncertainty and lack of harmonization might contribute to skepticism and vaccine hesitancy.
      Now, with the approval of the first vaccines, placebo-controlled trials are considered unethical
      • Wendler D
      • Ochoa J
      • Millum J
      • Grady C
      • Taylor HA
      COVID-19 vaccine trial ethics once we have efficacious vaccines.
      and alternative designs using sound causal inference methods are required such as, for example, noninferiority randomized trials
      • Fleming TR
      • Krause PR
      • Nason M
      • Longini IM
      • Henao-Restrepo AMM
      COVID-19 vaccine trials: the use of active controls and non-inferiority studies.
      or randomized vaccination rollout (similar to an individual stepped wedge design) when vaccines have been approved.

      Hemkens LG, Goodman SN. Randomized COVID-19 vaccination rollout can offer direct real-world evidence [e-pub ahead of print]. J Clin Epidemiol 2021. doi:10.1016/j.jclinepi.2021.05.014, accessed August 19, 2021.

      However, placebo-controlled trials might become again possible or even necessary in the future, if new, relevant questions arise (eg, the speculated need for a third dose of vaccination).

      Replication and meta-analysis

      Although large RCTs seem to be the solution in the COVID-19 pandemic, we have to be cautious to avoid claiming that they are always correct. Large RCTs can have their own shortfalls. For example, they might include multiple sites, some of which are inexperienced and thus get unreliable results affecting the final result for the entire trial.
      • Bristow MR
      • Silva Enciso J
      • Gersh BJ
      • et al.
      Detection and management of geographic disparities in the TOPCAT trial: lessons learned and derivative recommendations.
      Moreover, they might not be able to examine sources of potential heterogeneity (eg, different dose or timing of the intervention) that would be provided when the same sample size comes from several trials that are combined in a meta-analysis.
      It is thus useful to have more than just 1 large trial. For COVID-19, this was the case in many circumstances. The World Health Organization Solidarity and RECOVERY adaptive trials, for example, both assessed hydroxychloroquine and consistently showed no clinical benefit on 28-day mortality (rate ratio, 1.19; 95% confidence interval, 0.89-1.59 and rate ratio, 1.11; 95% confidence interval, 0.98- 1.26, respectively).
      COVID-19 has also led to rethinking the role, conduct, and dissemination of systematic reviews and meta-analyses. With the rapid evolution of the emerging evidence, many traditional systematic reviews and meta-analyses are at risk of being outdated before being published. Rapid reviews, scoping reviews, and living reviews are becoming increasingly popular;
      • Maguire BJ
      • Guérin PJ
      A living systematic review protocol for COVID-19 clinical trial registrations.
      • Fretheim A
      • Brurberg KG
      • Forland F
      Rapid reviews for rapid decision-making during the coronavirus disease (COVID-19) pandemic, Norway, 2020.
      • Siemieniuk RA
      • Bartoszko JJ
      • Ge L
      • et al.
      Drug treatments for COVID-19: living systematic review and network meta-analysis.

      Boutron I, Chaimani A, Meerpohl JJ, et al. Interventions for preventing and treating COVID-19: living systematic reviews and network meta-analyses. Available at: https://zenodo.org/record/4018607#.YDj_r5P0k1I. Accessed February 26, 2021.

      very large numbers of such evidence syntheses were undertaken and published very quickly.
      In addition, many systematic reviews have been conducted on observational studies often concluding that RCTs are needed to confirm the observed results. Such as, for example, the systematic reviews on physical distancing, face masks, and eye protection to prevent COVID-19, which provided evidence graded as “moderate certainty” for social distancing and “low certainty” for the others.
      • Chu DK
      • Akl EA
      • Duda S
      • et al.
      Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis.
      More robust randomized evidence is still lacking and is needed to confirm the observed associations.

      Adaptive designs

      The COVID-19 clinical research agenda was triggered by the urgent need to determine effective therapies; nevertheless, most potential therapies were evaluated in small 2-arm trials rendering the process extremely inefficient. Adaptive platform trials give the opportunity to simultaneously assess multiple interventions all the while allowing to drop or include new interventions as new evidence emerges.
      • Park JJ
      • Thorlund K
      • Mills EJ
      Critical concepts in adaptive clinical trials.
      Such designs aim to maximize flexibility in the trial without compromising its integrity and validity. By analyzing accumulating data through prespecified interim analyses, prompt adaptations can be made such as, for example, stopping early a treatment arm for superiority or futility and changing the randomization allocation ratio, in favour of the most promising treatment arms. The RECOVERY platform trial is probably the most illustrative example of a successful adaptive design

      RECOVERY. RECOVERY Protocol: Randomised evaluation of COVID-19 therapy. Available at: https://www.recoverytrial.net/files/recovery-protocol-v12-1-2020-12-16.pdf. Accessed January 27, 2021.

      (Table 2 and Box 2).
      Table 2Rate of dissemination
      TrialInterventionPress release datePreprint datePublication dateRegulatory aspects
      RECOVERYHydroxychloroquineJune 5, 2020July 15, 2020 (preliminary results)October 8, 2020The FDA issued an EUA on March 28, 2020, which was then revoked on June 15, 2020.

      The EMA recommended that it should only be used in clinical trials or national emergency use programs and issued reports on safety concerns.
      Lopinavir-ritonavirJune 29, 2020October 5, 2020
      AzithromycinDecember 14, 2020December 14, 2020 (preliminary results)February 2, 2021
      DexamethasoneJune 16, 2020June 22, 2020 (preliminary results)July 17, 2020 (preliminary results)

      February 25, 2021 (final report)
      The EMA endorsed the use of dexamethasone on September 18, 2020.

      The UK authorized the use of dexamethasone on June 19, 2020.
      Convalescent plasmaJanuary 15, 2021The FDA issued an EUA on August 23, 2020 for hospitalized COVID-19 patients.

      The EUA issued a revision on February 4, 2021, excluding low titer.
      TocilizumabFebruary 11, 2020
      NCT04292899

      (severe COVID-19)
      RemdesivirApril 29, 2020May 27, 2020The FDA issued an EUA on May 1, 2020 for severe COVID-19, expanded on August 28, 2020 to all hospitalized patients.

      On October 22, 2020, remdesivir was approved for adults and pediatric patients (12 years of age or older and weighing at least 40 kg) for the treatment of COVID-19 requiring hospitalization.

      The EMA granted a conditional marketing authorization on June 25, 2020.
      NCT04292730

      (moderate COVID-19)
      RemdesivirJune 1, 2020August 21, 2020
      NCT04368728Pfizer-BioNTech vaccine:

      BNT162b1 and BNT162b2
      Phase 3:

      November 9, 2020 (interim analyses - BNT162b2)

      November 18, 2020 (final efficacy analysis- BNT162b2)
      Phase 1/2:

      July 1, 2020 (interim report - BNT162b1)

      August 28, 2020 (preliminary results - BNT162b2)
      Phase 1/2:

      August 12, 2020 (interim report, BNT162b1 and BNT162b2)

      October 14, 2020 (safety results, BNT162b1 and BNT162b2)

      Phase 3:

      December 10, 2020 (safety and efficacy, BNT162b2)
      The FDA issued an EUA on December 11, 2020.

      The EMA granted conditional marketing authorization for BNT162b2 on December 21, 2020.
      NCT04470427Moderna vaccinePhase 1:

      May 18, 2020 (interim analysis)

      Phase 3:

      November 16, 2020 (primary end point)
      Phase 3:

      December 30, 2020
      The FDA issued an EUA on December 18, 2020.

      The EMA granted conditional marketing authorization on January 6, 2021.
      NCT04400838

      NCT04324606

      NCT04444674

      ISRCTN89951424
      AstraZeneca vaccineNovember 23, 2020 (primary end point pool analysis phase 2/3 and phase 3)Phase 2/3:

      February 4, 2021 (UK efficacy results)

      Phase 2/3 and 3:

      February 1, 2021 (efficacy booster dose)
      Phase 1/2:

      July 20, 2020 (preliminary report on safety)

      December 17, 2020 (exploratory analysis on the immune response)

      Phase 2/3 and 3:

      November 19, 2020 (UK preliminary results)

      December 8, 2020 (interim pooled analysis of 4 studies)

      February 19, 2021 (efficacy booster dose)
      The EMA granted conditional marketing authorization on January 29, 2021.

      In March 2021, several European countries decided to pause the vaccination because of events involving blood clots. The EMA led an investigation and maintained its position that the benefits outweigh the risk of side effects.
      NCT04509947

      NCT04436276

      NCT04505722

      NCT04535453

      NCT04614948

      NCT04765384
      Janssen vaccinePhase 1/2a:

      September 25, 2020 (interim analysis)

      Phase 3:

      January 29, 2021 (interim analysis)
      Phase 1/2a:

      September 25, 2020 (interim analysis)
      Phase 1/2a:

      January 13, 2021 (interim analysis)

      March 11, 2020 (results from a unique centre)

      Phase 3:

      April 21, 2021 (primary analysis)
      The FDA issued an EUA on February 27, 2021.

      The EMA granted conditional marketing authorization on March 11, 2021.

      In April 2021, the FDA recommended pausing the vaccination because of rare blood clots events. After review of the cases, the FDA lifted the recommended pause.
      NCT04436471 NCT04437875

      NCT04530396
      Sputnik vaccine (Gamaleya Research Institute of Epidemiology and Microbiology, Health Ministry of the Russian Federation, Moscow, Russia)November 11, 2020 (interim phase 3 efficacy)

      December 14, 2020 (efficacy analysis)
      September 4, 2020 (phase 1/2)

      February 2, 2021 (interim phase 3 efficacy)

      Approved by the Russian Federation on August 11, 2020.
      EMA, European Medicines Agency; EUA, Emergency Use Authorization; FDA, US Food and Drug Administration; RECOVERY, Randomised Evaluation of COVID-19 Therapy; UK, United Kingdom.
      Key features of the RECOVERY trial
      Tabled 1
      • Platform trial assessing multiple intervention arms with a factorial design (ie, a patient can receive more than 1 experimental treatment).
      • Adaptive design allowing to add or drop intervention arms as evidence emerges.
      • Simplified online randomization easing the work of investigational sites.
      • Pragmatic features with broad eligibility criteria (ie, hospitalized, confirmed, or suspected SARS-CoV-2 infection, and no specific contraindication to their participation).
      • Data on primary end point, all-cause mortality at 28 days, are collected through a unique online follow-up form filled out at death, discharge, or at day 28, whichever comes first.
      • Linkage to national health registries to make use of routinely collected data.
      • As of February 2021, more than 36,000 patients have been recruited in more than 180 investigational sites (of which 2 are from outside of the United Kingdom).
      Other trials were originally designed for other conditions. For example, the Randomised, Embedded, Multi-factorial, Adaptive Platform Trial for Community-Acquired Pneumonia (REMAP-CAP) trial was designed

      REMAP-CAP. A Randomised, Embedded, Multi-factorial, Adaptive Platform Trial for Community-Acquired Pneumonia. Available at: https://www.remapcap.org. Accessed February 26, 2021.

      as an adaptive platform trial to assess interventions for community-acquired pneumonia. As early as March 2020, they adapted their protocol to include patients with COVID-19 and to assess COVID-19-relevant interventions.

      REMAP-CAP. REMAP-CAP Response to the COVID-19 Pandemic. Available at: https://www.remapcap.org/coronavirus. Accessed February 26, 2021.

      In the 2814 registered RCTs in COVID-evidence, 15 were registered before January 2020 but then included patients with COVID-19.

      Dissemination, transparency, data-sharing

      COVID-19 is also characterized by an unprecedented volume of research being quickly disseminated in the scientific literature and preprint servers with over 200,000 items published by early December 2020.
      • Else H
      How a torrent of COVID science changed research publishing—in seven charts.
      For example, the JAMA Network recorded 53% more submissions in the first quarter of 2020 compared with the year before
      • Kwon D
      How swamped preprint servers are blocking bad coronavirus research.
      and medRxiv saw its median number of daily submission increasing by > 8-fold from before the pandemic (median, 6; interquartile range, 4-8) compared with during the pandemic (median, 51; interquartile range, 23-83) reaching a total of 7695 posted articles between July 2019 and June 2020.
      • Krumholz HM
      • Bloom T
      • Sever R
      • et al.
      Submissions and downloads of preprints in the first year of medRxiv.
      The publication process was drastically shorter for COVID-related articles ranging from 6 to 20 days from submission to publication vs > 100 days for non-COVID-19 related articles.
      • Palayew A
      • Norgaard O
      • Safreed-Harmon K
      • et al.
      Pandemic publishing poses a new COVID-19 challenge.
      • Putman MS
      • Ruderman EM
      • Niforatos JD
      Publication rate and journal review time of COVID-19–related research.
      • Horbach SPJM
      Pandemic publishing: medical journals drastically speed up their publication process for Covid-19.
      This exciting contribution to the scientific literature by the mobilized scientific community has been tarnished by research-publishing scandals

      Retraction Watch. Marcus A. Lancet, NEJM retract controversial COVID-19 studies based on Surgisphere data. Available at: https://retractionwatch.com/2020/06/04/lancet-retracts-controversial-hydroxychloroquine-study. Accessed February 26, 2021.

      with > 70 COVID-19 related-article retractions recorded as of December 2020 on Retraction Watch.

      Retraction Watch. List of Retracted COVID-19 Papers Grows Past 70. Available at: https://retractionwatch.com/2020/12/30/list-of-retracted-covid-19-papers-grows-past-70. Accessed February 25, 2021.

      However, it is too early to assess the true effect of COVID-19-related retractions compared with other scientific research areas.

      Retraction Watch. List of Retracted COVID-19 Papers Grows Past 70. Available at: https://retractionwatch.com/2020/12/30/list-of-retracted-covid-19-papers-grows-past-70. Accessed February 25, 2021.

      Concerns have been raised regarding the quality of the publications being disseminated with most not presenting original data (ie, expert opinion pieces) and when original data were presented > 80% showed intermediate to high risk of bias and included small numbers of patients.
      • Raynaud M
      • Zhang H
      • Louis K
      • et al.
      COVID-19-related medical research: a meta-research and critical appraisal.
      Many journals have experienced a massive increase in submissions putting heightened pressure on editors and peer reviewers. Many suboptimal reports might pass through thinned-out review filters

      Oxford Academic. No Time for That Now! Qualitative Changes in Manuscript Peer Review During the Covid-19 Pandemic. Available at: https://academic.oup.com/rev/advance-article/doi/10.1093/reseval/rvaa037/6064166. Accessed April 26, 2021.

      and/or because they fit favoured narratives. In addition, dissemination of misinformation has been exacerbated by media attention and mob behaviour in social media platforms
      • Krittanawong C
      • Narasimhan B
      • Virk HUH
      • et al.
      Misinformation dissemination in Twitter in the COVID-19 Era.
      relayed by decision-makers and politicians.
      The rapidly evolving and fragmented landscape of the COVID-19 pandemic requires good research practice, reproducibility, and transparency to ensure informed decision-making and maintain public trust.
      • Ewers M
      • Ioannidis JPA
      • Plesnila N
      Access to data from clinical trials in the COVID-19 crisis: open, flexible, and time-sensitive.
      Data sharing allows reanalysis, synthesis, and building on existing evidence, increasing the reproducibility and transparency of the scientific method. Although those advantages of data sharing are well known and have been endorsed by the International Committee of Medical Journal Editors, the gap between the intent of sharing data and the actual availability of the data remains wide
      • Danchev V
      • Min Y
      • Borghi J
      • Baiocchi M
      • Ioannidis JPA
      Evaluation of data sharing after implementation of the International Committee of Medical Journal Editors Data Sharing Statement requirement.
      and is unlikely to be bridged during the COVID-19 pandemic. Several initiatives are under way to provide clinical trial data sharing platforms,
      • Li R
      • Wood J
      • Baskaran A
      • et al.
      Timely access to trial data in the context of a pandemic: the time is now.
      prospectively pool individual patient data from trials,
      • Petkova E
      • Antman EM
      • Troxel AB
      Pooling data from individual clinical trials in the COVID-19 era.
      and conduct collaborative meta-analyses of all available data.
      • Axfors C
      • Schmitt AM
      • Janiaud P
      • et al.
      Mortality outcomes with hydroxychloroquine and chloroquine in COVID-19 from an international collaborative meta-analysis of randomized trials.

      Regulatory issues

      Regulatory authorities throughout the world have been under immense pressure to make decisions as the evidence appeared, sometimes regardless of the source of the evidence and its credibility. For example, press releases of trial results have had a substantial effect with the FDA revoking the EUA for hydroxychloroquine just 10 days after or dexamethasone being recommended by the United Kingdom authorities just 3 days after the press releases from RECOVERY (Table 2).
      Up to April 2020, the FDA had issued 174 EUAs; most were for diagnostic tests for various disease outbreaks and only 3 had been issued in the past for therapeutics, which were all for the treatment of the 2009 influenza A (H1N1). Currently there are 6 EUAs for therapeutics and 3 EUAs for vaccines for COVID-19
      • Zhai MZ
      • Lye CT
      • Kesselheim AS
      Need for transparency and reliable evidence in Emergency Use Authorizations for coronavirus disease 2019 (COVID-19) therapies.
      but no COVID-19 therapeutics have yet received a full or accelerated approval from the FDA. EUAs are crucial in a public health crisis but they should not compromise the integrity of the scientific method.

      Doing trials under pandemic disruption

      Conducting non-COVID-19 trials during the pandemic has been challenging:
      • Mitchell EJ
      • Ahmed K
      • Breeman S
      • et al.
      It is unprecedented: trial management during the COVID-19 pandemic and beyond.
      the lockdown and social distancing measures directly interfere with efficient enrollment and randomization;
      • McDermott MM
      • Newman AB
      Preserving clinical trial integrity during the coronavirus pandemic.
      and hospitals overwhelmed because of COVID-19 cases might have redirected their resources away from clinical research. As early as March 2020, the FDA had issued guidance for conducting trials during the pandemic

      US Food & Drug Administration. FDA Guidance on Conduct of Clinical Trials of Medical Products During the COVID-19 Public Health Emergency. Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/fda-guidance-conduct-clinical-trials-medical-products-during-covid-19-public-health-emergency. Accessed February 25, 2021.

      ensuring patient safety all the while minimizing the effect on the conduct of trials not related to COVID-19. Yet, from January to March 2020, an average of 1147 non-COVID-19 trials were stopped (ie, recruiting status changed to active, not recruiting, suspended, completed, terminated, or withdrawn) per month vs 638 trials per month in 2019.
      • Gaudino M
      • Arvind V
      • Hameed I
      • et al.
      Effects of the COVID-19 pandemic on active non-COVID clinical trials.
      The peak of suspended trials explicitly due to COVID-19 was reached in April 2020 with 1021 trials being suspended.
      • Xue JZ
      • Smietana K
      • Poda P
      • et al.
      Clinical trial recovery from COVID-19 disruption.
      Although the non-COVID-19 clinical research agenda was disrupted, this experience has also emphasised the need to implement innovative solutions such as online recruitment and informed consent,
      • Mitchell EJ
      • Ahmed K
      • Breeman S
      • et al.
      It is unprecedented: trial management during the COVID-19 pandemic and beyond.
      streamlining interactions between stakeholders (funders, site staff, trial manager, and steering and data monitoring committees [DMCs]) including remote monitoring of clinical trials and shorter regulatory processes (detailed in the section: Doing Trials Differently in the Future), remote delivery of interventions using mail drop-off of pills or devices,
      • McDermott MM
      • Newman AB
      Remote research and clinical trial integrity during and after the coronavirus pandemic.
      and collection of outcomes data using electronic health records. Some of these lessons and gained skills might be useful to implement also in the post-pandemic world and might enhance the efficiency of the clinical research enterprise at large.

      Doing Trials Differently in the Future

      The COVID-19 situation highlighted a number of excellent approaches to overcome research challenges in the pandemic. Many of them represent promising solutions for the most common perceived limitations of RCTs in general.

      Planning

      Putting research in context is critical to increase research value and return of investment. Instead of separately planning small trials, joining forces and contributing to large trials is key. This requires that such large trials are identifiable and open for collaboration. Because large collaborative studies acquire great prestige, visibility, and citations, they might offer a stronger incentive for investigators to join them rather than perform their own underpowered study.
      Platforms such as the COVID-evidence project aim to show the trial research agenda on a website and actively bring together research groups and trial teams to collaborate.

      Janiaud P, Axfors C, Saccilotto R, Hemkens L. COVID-evidence: a living database of trials on interventions for COVID-19. Available at: https://osf.io/gehfx. Accessed March 1, 2021.

      Such trials need to be expandable (ie, designed in a way that centres can easily be added), to lower the bar for collaboration and contribution. The RECOVERY trial had such expandability and started with > 130 hospitals recruiting 1000 patients in 2 weeks, and 176 United Kingdom hospitals had then recruited 10,000 patients in 2 months.

      RECOVERY. RECOVERY Trial Rolled Out Across the UK. Available at: https://www.recoverytrial.net/news/update. Accessed March 2, 2021.

      Approximately 1 year later, in February 2021, it even expanded to a different continent with centres in Nepal and Indonesia, then having recruited more than 36,000 patients globally.

      RECOVERY. World's Largest Clinical Trial for COVID-19 Treatments Expands Internationally. Available at: https://www.recoverytrial.net/news/world2019s-largest-clinical-trial-for-covid-19-treatments-expands-internationally. Accessed February 26, 2021.

      A critical issue is the avoidance of creating an environment in which trials compete for the same patients (ie, 2 trials conducted in the same location with similar eligibility criteria). Trials often use the exclusion criterion “enrollment in another clinical trial.” However, this needs to be carefully considered when planning a clinical trial and allowing patients to participate in a second treatment protocol might be desirable when no other treatments are available.
      • Gertner E
      COVID-19 trial co-enrolment and subsequent enrolment.
      Centres with limited research expertise might not be able to perform rigourous research under acute circumstances when there is limited or no sufficient time and resources for training and for auditing centre performance. Careful planning of included centres should be done upstream because inclusion of inexperienced, poorly performing centres might affect the overall trial results, as has been shown in some examples in the pre-COVID era.
      • Bristow MR
      • Silva Enciso J
      • Gersh BJ
      • et al.
      Detection and management of geographic disparities in the TOPCAT trial: lessons learned and derivative recommendations.

      Design and analysis

      The COVID-19 trial research agenda has clearly shown that RCTs can be launched very quickly. The setup of novel data infrastructures is time-consuming but can often be avoided. Using available infrastructures, either from existing clinical trials, cohorts, registries, or from routine care can be extremely helpful to rapidly generate evidence. The REMAP-CAP trial used its existing trial processes and infrastructures

      REMAP-CAP. REMAP-CAP Response to the COVID-19 Pandemic. Available at: https://www.remapcap.org/coronavirus. Accessed February 26, 2021.

      and has been expanded for COVID-19 patients. The RECOVERY trial used nationwide electronic health record data for outcome assessments and combined this with short-term active data collection.

      RECOVERY. RECOVERY Protocol: Randomised evaluation of COVID-19 therapy. Available at: https://www.recoverytrial.net/files/recovery-protocol-v12-1-2020-12-16.pdf. Accessed January 27, 2021.

      A trial on mental health interventions to cope with COVID-19 has been embedded in a cohort of patients with sclerodermia.
      • Thombs BD
      • Kwakkenbos L
      • Carrier ME
      • et al.
      Protocol for a partially nested randomised controlled trial to evaluate the effectiveness of the scleroderma patient-centered intervention network COVID-19 home-isolation activities together (SPIN-CHAT) program to reduce anxiety among at-risk scleroderma patients.
      Future pandemic research preparedness might include the setup of very large, nationwide cohorts in which multiple interventions could be tested as in Trials Within Cohorts (TWiCs).
      • Kim SY
      • Flory J
      • Relton C
      Ethics and practice of trials within cohorts: an emerging pragmatic trial design.
      Several trials showed how clinical research can be embedded in routine clinical care, becoming even the standard of care. The RECOVERY trial with its 3 inclusion criteria and streamlined recruitment processes avoided artificial research settings by embedding the trial in routine care and ensuring inclusivity of the recruitment. One of 6 patients hospitalized for COVID-19 in the United Kingdom took part in RECOVERY.

      Kai Kupferschmidt. One U.K. trial is transforming COVID-19 treatment. Why haven't others delivered more results?. Sciencemag.org. July 2, 2020. Available at:https://www.sciencemag.org/news/2020/07/one-uk-trial-transforming-covid-19-treatment-why-haven-t-others-delivered-more-results. Accessed March 2, 2021.

      This resembled real-world treatment choices and not only boosted trial feasibility—by design, it also maximized the applicability of the results. Similar highly pragmatic trials that are embedded in clinical routine might systematically evaluate treatment strategies, ideally for all relevant health care choices also beyond COVID-19.

      Streamlined administration and collaborations

      Funders, institutional review boards (ie, ethics committees), DMCs, and regulators have shown remarkable adaptation and flexibility when confronted with the urgency of the pandemic. Many countries have implemented a fast-track procedure for the authorization of clinical trials including ethical reviews.

      ECRIN - European Clinical Research Infrastructure Network. Fast track Procedure for COVID 19 Clinical Trials. Available at: https://ecrin.org/fast-track-procedure-covid-19-clinical-trials. Accessed February 26, 2021.

      In the United Kingdom the Health Research Authority reduced the ethic review process from 60 to 10 days.
      • Mather N
      How we accelerated clinical trials in the age of coronavirus.
      In Switzerland, the Swiss Association of Research Ethics Committees lists all approved and submitted but not yet approved trials, promoting transparency of their processes.

      Swissethics. Swiss Association of Research Ethics Committees. Information on the Coronavirus: In Order to Promote Transparency and Facilitate Research Coordination, Swissethics Is Now Also Publishing Research Projects That Have Been Submitted But Not Yet Approved. Available at: https://swissethics.ch/en/news/2020/03/30/informationen-zum-corona-virus-um-die-transparenz-zu-foerdern-und-die-forschungskoordination-zu-erleichtern-veroeffentlicht-swissethics-neu-auch-eingereichte-aber-noch-nicht-bewilligte-forschungsprojekte. Accessed February 26, 2021.

      DMCs play a crucial role in the successful conduct of adaptive platform trials by reviewing interim analyses and safety data, and making recommendation on the fate of trials (or treatment-specific arms). Trials are often underpowered to assess safety; better coordination is needed between the DMCs of trials addressing similar clinical questions, allowing them to share emerging evidence to better assess the risk and benefit of interventions.
      • DeMets DL
      • Fleming TR
      Achieving effective informed oversight by DMCs in COVID clinical trials.
      The lack of coordination and collaboration raises important ethical issues to the detriment of patients but also to the medical staff already overwhelmed.
      • Bompart F
      Ethical rationale for better coordination of clinical research on COVID-19.
      Higher authorities such as the World Health Organization with the Solidarity trial or the United Kingdom government call to conduct a trial to investigate dosing of alternating vaccines,

      University of Oxford. Oxford Leads First Trial Investigating Dosing With Alternating Vaccines. Available at: https://www.ox.ac.uk/news/2021-02-04-oxford-leads-first-trial-investigating-dosing-alternating-vaccines. Accessed February 26, 2021.

      have shown their ability to set priorities and foster nation- and worldwide collaborations.
      The pandemic has shed light on the need and feasibility to streamline the excessively complex administrative procedures that have burdened clinical research. Stakeholders need close interoperability allowing for data and resources flow between stakeholders nationally and internationally (ie, regulatory authorities, funders, clinical sites, and monitoring committees).
      • Angus DC
      • Gordon AC
      • Bauchner H
      Emerging lessons from COVID-19 for the US clinical research enterprise.

      Conclusion

      Overall, COVID-19 has clearly challenged the traditional myths that conducting clinical trials must be difficult and time-consuming, resulting in trials that are too small or take too long to be completed. This crisis has clearly magnified those shortcomings but has also shown how to make clinical research more useful—by doing trials with elegant designs reduced to essential elements and prepared to be combined with others, allowing them to be performed with speed and flexibility in large collaborations.

      Funding Sources

      The Meta-Research Innovation Center at Stanford (METRICS), Stanford University, is supported by a grant from the Laura and John Arnold Foundation. The work of John Ioannidis is supported by an unrestricted gift from Sue and Bob O'Donnell. COVID-evidence is supported by the Swiss National Science Foundation (31CA30_196190). The funders had no role in the study design; in the collection, analysis, or interpretation of data; in the writing of the report; or in the decision to submit the report for publication.

      Disclosures

      The authors have no conflicts of interest to disclose.

      References

        • Janiaud P
        • Axfors C
        • Van't Hooft J
        • et al.
        The worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days.
        F1000Res. 2020; 9: 1193
        • Janiaud P
        • Axfors C
        • Ioannidis JPA
        • Hemkens LG
        Recruitment and results reporting of COVID-19 randomized clinical trials registered in the first 100 days of the pandemic.
        JAMA Netw Open. 2021; 4e210330
        • Cristea IA
        • Naudet F
        • Ioannidis JPA
        Preserving equipoise and performing randomised trials for COVID-19 social distancing interventions.
        Epidemiol Psychiatr Sci. 2020; 29: e184
      1. Janiaud P, Axfors C, Saccilotto R, Hemkens L. COVID-evidence: a living database of trials on interventions for COVID-19. Available at: https://osf.io/gehfx. Accessed March 1, 2021.

        • Song G
        • Cheng MQ
        • Wei XW
        Analysis of the WHO ICTRP for novel coronavirus clinical trial registrations.
        Medicine (Baltimore). 2020; 99: e22840
        • Maguire BJ
        • McLean ARD
        • Rashan S
        • et al.
        Baseline results of a living systematic review for COVID-19 clinical trial registrations.
        Wellcome Open Res. 2020; 5: 116
        • Karlsen APH
        • Wiberg S
        • Laigaard J
        • et al.
        A systematic review of trial registry entries for randomized clinical trials investigating COVID-19 medical prevention and treatment.
        PLoS One. 2020; 15e0237903
        • Batta A
        • Khirasaria R
        • Kapoor V
        • Varshney D
        Therapeutic clinical trials to combat COVID-19 pandemic in India: analysis from trial registry.
        J Basic Clin Physiol Pharmacol. 2020; 32: 267-278
        • Wang Y
        • Zhou Q
        • Xu M
        • Kang J
        • Chen Y
        Characteristics of clinical trials relating to COVID-19 registered at ClinicalTrials.gov.
        J Clin Pharm Ther. 2020; 45: 1357-1362
        • Lythgoe MP
        • Middleton P
        Ongoing clinical trials for the management of the COVID-19 pandemic.
        Trends Pharmacol Sci. 2020; 41: 363-382
        • Glasziou PP
        • Sanders S
        • Hoffmann T
        Waste in COVID-19 research.
        BMJ. 2020; 369: m1847
        • Collaborative Group RECOVERY
        • Horby P
        • Mafham M
        • et al.
        Effect of hydroxychloroquine in hospitalized patients with COVID-19.
        N Engl J Med. 2020; 383: 2030-2040
        • Axfors C
        • Schmitt AM
        • Janiaud P
        • et al.
        Mortality outcomes with hydroxychloroquine and chloroquine in COVID-19 from an international collaborative meta-analysis of randomized trials.
        Nat Commun. 2021; 12: 2349
        • RECOVERY
        Statement from the Chief Investigators of the Randomised Evaluation of COVid-19 thERapY (RECOVERY) Trial on hydroxychloroquine.
        5 June 2020 (No clinical benefit from use of hydroxychloroquine in hospitalised patients with COVID-19. Available at:) (Accessed March 1, 2021)
      2. FDA. US Food & Drug Administration. Emergency Use Authorization. Available at:https://www.fda.gov/emergency-preparedness-and-response/mcm-legal-regulatory-and-policy-framework/emergency-use-authorization. Accessed February 26, 2021.

      3. European Medicines Agency. Treatments and vaccines for COVID-19. Available at: https://www.ema.europa.eu/en/human-regulatory/overview/public-health-threats/coronavirus-disease-covid-19/treatments-vaccines-covid-19. Accessed February 26, 2021.

        • Singh TU
        • Parida S
        • Lingaraju MC
        • et al.
        Drug repurposing approach to fight COVID-19.
        Pharmacol Rep. 2020; 72: 1479-1508
        • Rodgers F
        • Pepperrell T
        • Keestra S
        • Pilkington V
        Missing clinical trial data: the evidence gap in primary data for potential COVID-19 drugs.
        Trials. 2021; 22: 59
        • Desai A
        • Gyawali B
        End points used in phase III randomized controlled trials of treatment options for COVID-19.
        EClinicalMedicine. 2020; 23100403
        • Dodd LE
        • Follmann D
        • Wang J
        • et al.
        End points for randomized controlled clinical trials for COVID-19 treatments.
        Clin Trials. 2020; 17: 472-482
        • Zarin DA
        • Rosenfeld S
        Lack of harmonization of coronavirus disease ordinal scales.
        Clinical Trials. 2021; 18: 263-264
        • COMET Initiative
        Core outcome set developers’ response to COVID-19.
        April 2021 (Available at:) (Accessed February 26, 2021)
        • Prendki V
        • Tau N
        • Avni T
        • et al.
        A systematic review assessing the under-representation of elderly adults in COVID-19 trials.
        BMC Geriatr. 2020; 20: 538
        • Ioannidis JPA
        • Axfors C
        • Contopoulos-Ioannidis DG
        Population-level COVID-19 mortality risk for non-elderly individuals overall and for non-elderly individuals without underlying diseases in pandemic epicenters.
        Environ Res. 2020; 188109890
        • Horby PW
        • Mafham M
        • Bell JL
        • et al.
        Lopinavir–ritonavir in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial.
        Lancet. 2020; 396: 1345-1352
        • The RECOVERY Collaborative Group
        Dexamethasone in hospitalized patients with Covid-19—preliminary report.
        N Engl J Med. 2021; 384: 693-704
        • Beers E
        • Moerkerken DC
        • Leufkens HGM
        • Egberts TCG
        • Jansen PAF
        Participation of older people in preauthorization trials of recently approved medicines.
        J Am Geriatr Soc. 2014; 62: 1883-1890
        • Ruiter R
        • Burggraaf J
        • Rissmann R
        Under-representation of elderly in clinical trials: an analysis of the initial approval documents in the Food and Drug Administration database.
        Br J Clin Pharmacol. 2019; 85: 838-844
        • Ioannidis JPA
        • Axfors C
        • Contopoulos-Ioannidis DG
        Second versus first wave of COVID-19 deaths: shifts in age distribution and in nursing home fatalities.
        Environ Res. 2021; 195110856
        • Chokkara S
        • Volerman A
        • Ramesh S
        • Laiteerapong N
        Examining the inclusivity of US trials of COVID-19 treatment.
        J Gen Intern Med. 2021; 36: 1443-1445
        • Bhopal SS
        • Bhopal R
        Sex differential in COVID-19 mortality varies markedly by age.
        Lancet. 2020; 396: 532-533
        • Schiffer VMMM
        • Janssen EBNJ
        • van Bussel BCT
        • et al.
        The “sex gap” in COVID-19 trials: a scoping review.
        EClinicalMedicine. 2020; 29100652
        • Bundgaard H
        • Bundgaard JS
        • Raaschou-Pedersen DET
        • et al.
        Effectiveness of adding a mask recommendation to other public health measures to prevent SARS-CoV-2 infection in Danish mask wearers: a randomized controlled trial.
        Ann Intern Med. 2021; 174: 335-343
        • Fretheim A
        • Flatø M
        • Steens A
        • et al.
        COVID-19: we need randomised trials of school closures.
        J Epidemiol Community Health. 2020; 74: 1078-1079
        • Glasziou P
        • Altman DG
        • Bossuyt P
        • et al.
        Reducing waste from incomplete or unusable reports of biomedical research.
        Lancet. 2014; 383: 267-276
        • Yusuf S
        • Collins R
        • Peto R
        Why do we need some large, simple randomized trials?.
        Stat Med. 1984; 3: 409-420
      4. ISIS-4: a randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction. ISIS-4 (Fourth International Study of Infarct Survival) Collaborative Group.
        Lancet. 1995; 345: 669-685
        • Hemkens LG
        • Contopoulos-Ioannidis DG
        • Ioannidis JPA
        Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey.
        BMJ. 2016; 352: i493
        • Ioannidis JPA
        Contradicted and initially stronger effects in highly cited clinical research.
        JAMA. 2005; 294: 218-228
        • Mahase E
        Covid-19: US approves emergency use of convalescent plasma despite warnings over lack of evidence.
        BMJ. 2020; 370: m3327
        • Joyner MJ
        • Senefeld JW
        • Klassen SA
        • et al.
        Effect of convalescent plasma on mortality among hospitalized patients with COVID-19: initial three-month experience.
        medRxiv. 2020; (Preprint posted online August 12)https://doi.org/10.1101/2020.08.12.20169359
      5. RECOVERY. Convalescent Plasma results—RECOVERY Trial. Available at: https://www.recoverytrial.net/results/convalescent-plasma. Accessed February 26, 2021.

      6. Mehra MR, Desai SS, Ruschitzka F, Patel AN. RETRACTED: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis [e-pub ahead of print] [erratum in: Lancet https://doi:org/10.1016/S0140-6736(20)31249-6] [retraction in: Lancet https://doi:org/10.1016/S0140-6736(20)31324-6]. Lancet. doi:10.1016/S0140-6736(20)31180-6, Accessed February 26, 2021.

      7. Retraction Watch. Marcus A. Lancet, NEJM retract controversial COVID-19 studies based on Surgisphere data. Available at: https://retractionwatch.com/2020/06/04/lancet-retracts-controversial-hydroxychloroquine-study. Accessed February 26, 2021.

        • Carlisle JB
        False individual patient data and zombie randomised controlled trials submitted to Anaesthesia.
        Anaesthesia. 2021; 76: 472-479
        • Ioannidis JPA
        Hundreds of thousands of zombie randomised trials circulate among us.
        Anaesthesia. 2021; 76: 444-447
      8. European Medicines Agency. EMA's safety committee continues investigation of COVID-19 Vaccine AstraZeneca and thromboembolic events – further update. Available at: https://www.ema.europa.eu/en/news/emas-safety-committee-continues-investigation-covid-19-vaccine-astrazeneca-thromboembolic-events. Accessed February 26, 2021.

      9. US Food & Drug Administration. Joint CDC and FDA Statement on Johnson & Johnson COVID-19 Vaccine. Available at: https://www.fda.gov/news-events/press-announcements/joint-cdc-and-fda-statement-johnson-johnson-covid-19-vaccine. Accessed February 26, 2021.

        • Wendler D
        • Ochoa J
        • Millum J
        • Grady C
        • Taylor HA
        COVID-19 vaccine trial ethics once we have efficacious vaccines.
        Science. 2020; 370: 1277-1279
        • Fleming TR
        • Krause PR
        • Nason M
        • Longini IM
        • Henao-Restrepo AMM
        COVID-19 vaccine trials: the use of active controls and non-inferiority studies.
        Clin Trials. 2021; 18: 335-342
      10. Hemkens LG, Goodman SN. Randomized COVID-19 vaccination rollout can offer direct real-world evidence [e-pub ahead of print]. J Clin Epidemiol 2021. doi:10.1016/j.jclinepi.2021.05.014, accessed August 19, 2021.

        • Bristow MR
        • Silva Enciso J
        • Gersh BJ
        • et al.
        Detection and management of geographic disparities in the TOPCAT trial: lessons learned and derivative recommendations.
        JACC Basic Transl Sci. 2016; 1: 180-189
        • Maguire BJ
        • Guérin PJ
        A living systematic review protocol for COVID-19 clinical trial registrations.
        Wellcome Open Res. 2020; 5: 60
        • Fretheim A
        • Brurberg KG
        • Forland F
        Rapid reviews for rapid decision-making during the coronavirus disease (COVID-19) pandemic, Norway, 2020.
        Eurosurveillance. 2020; 252000687
        • Siemieniuk RA
        • Bartoszko JJ
        • Ge L
        • et al.
        Drug treatments for COVID-19: living systematic review and network meta-analysis.
        BMJ. 2020; 370: m2980
      11. Boutron I, Chaimani A, Meerpohl JJ, et al. Interventions for preventing and treating COVID-19: living systematic reviews and network meta-analyses. Available at: https://zenodo.org/record/4018607#.YDj_r5P0k1I. Accessed February 26, 2021.

        • Chu DK
        • Akl EA
        • Duda S
        • et al.
        Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis.
        Lancet. 2020; 395: 1973-1987
        • Park JJ
        • Thorlund K
        • Mills EJ
        Critical concepts in adaptive clinical trials.
        Clin Epidemiol. 2018; 10: 343-351
      12. RECOVERY. RECOVERY Protocol: Randomised evaluation of COVID-19 therapy. Available at: https://www.recoverytrial.net/files/recovery-protocol-v12-1-2020-12-16.pdf. Accessed January 27, 2021.

      13. REMAP-CAP. A Randomised, Embedded, Multi-factorial, Adaptive Platform Trial for Community-Acquired Pneumonia. Available at: https://www.remapcap.org. Accessed February 26, 2021.

      14. REMAP-CAP. REMAP-CAP Response to the COVID-19 Pandemic. Available at: https://www.remapcap.org/coronavirus. Accessed February 26, 2021.

        • Else H
        How a torrent of COVID science changed research publishing—in seven charts.
        Nature. 2020; 588: 553
        • Kwon D
        How swamped preprint servers are blocking bad coronavirus research.
        Nature. 2020; 581: 130-131
        • Krumholz HM
        • Bloom T
        • Sever R
        • et al.
        Submissions and downloads of preprints in the first year of medRxiv.
        JAMA. 2020; 324: 1903-1905
        • Palayew A
        • Norgaard O
        • Safreed-Harmon K
        • et al.
        Pandemic publishing poses a new COVID-19 challenge.
        Nat Hum Behav. 2020; 4: 666-669
        • Putman MS
        • Ruderman EM
        • Niforatos JD
        Publication rate and journal review time of COVID-19–related research.
        Mayo Clin Proc. 2020; 95: 2290-2291
        • Horbach SPJM
        Pandemic publishing: medical journals drastically speed up their publication process for Covid-19.
        Quantitative Science Studies. 2020; 1: 1056-1067
      15. Retraction Watch. List of Retracted COVID-19 Papers Grows Past 70. Available at: https://retractionwatch.com/2020/12/30/list-of-retracted-covid-19-papers-grows-past-70. Accessed February 25, 2021.

        • Raynaud M
        • Zhang H
        • Louis K
        • et al.
        COVID-19-related medical research: a meta-research and critical appraisal.
        BMC Med Res Methodol. 2021; 21: 1
      16. Oxford Academic. No Time for That Now! Qualitative Changes in Manuscript Peer Review During the Covid-19 Pandemic. Available at: https://academic.oup.com/rev/advance-article/doi/10.1093/reseval/rvaa037/6064166. Accessed April 26, 2021.

        • Krittanawong C
        • Narasimhan B
        • Virk HUH
        • et al.
        Misinformation dissemination in Twitter in the COVID-19 Era.
        Am J Med. 2020; 133: 1367-1369
        • Ewers M
        • Ioannidis JPA
        • Plesnila N
        Access to data from clinical trials in the COVID-19 crisis: open, flexible, and time-sensitive.
        J Clin Epidemiol. 2021; 130: 143-146
        • Danchev V
        • Min Y
        • Borghi J
        • Baiocchi M
        • Ioannidis JPA
        Evaluation of data sharing after implementation of the International Committee of Medical Journal Editors Data Sharing Statement requirement.
        JAMA Netw Open. 2021; 4e2033972
        • Li R
        • Wood J
        • Baskaran A
        • et al.
        Timely access to trial data in the context of a pandemic: the time is now.
        BMJ Open. 2020; 10e039326
        • Petkova E
        • Antman EM
        • Troxel AB
        Pooling data from individual clinical trials in the COVID-19 era.
        JAMA. 2020; 324: 543
        • Zhai MZ
        • Lye CT
        • Kesselheim AS
        Need for transparency and reliable evidence in Emergency Use Authorizations for coronavirus disease 2019 (COVID-19) therapies.
        JAMA Intern Med. 2020; 180: 1145-1146
        • Mitchell EJ
        • Ahmed K
        • Breeman S
        • et al.
        It is unprecedented: trial management during the COVID-19 pandemic and beyond.
        Trials. 2020; 21: 784
        • McDermott MM
        • Newman AB
        Preserving clinical trial integrity during the coronavirus pandemic.
        JAMA. 2020; 323: 2135-2136
      17. US Food & Drug Administration. FDA Guidance on Conduct of Clinical Trials of Medical Products During the COVID-19 Public Health Emergency. Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/fda-guidance-conduct-clinical-trials-medical-products-during-covid-19-public-health-emergency. Accessed February 25, 2021.

        • Gaudino M
        • Arvind V
        • Hameed I
        • et al.
        Effects of the COVID-19 pandemic on active non-COVID clinical trials.
        J Am Coll Cardiol. 2020; 76: 1605-1606
        • Xue JZ
        • Smietana K
        • Poda P
        • et al.
        Clinical trial recovery from COVID-19 disruption.
        Nat Rev Drug Discov. 2020; 19: 662-663
        • McDermott MM
        • Newman AB
        Remote research and clinical trial integrity during and after the coronavirus pandemic.
        JAMA. 2021; 325: 1935-1936
      18. RECOVERY. RECOVERY Trial Rolled Out Across the UK. Available at: https://www.recoverytrial.net/news/update. Accessed March 2, 2021.

      19. RECOVERY. World's Largest Clinical Trial for COVID-19 Treatments Expands Internationally. Available at: https://www.recoverytrial.net/news/world2019s-largest-clinical-trial-for-covid-19-treatments-expands-internationally. Accessed February 26, 2021.

        • Gertner E
        COVID-19 trial co-enrolment and subsequent enrolment.
        Lancet. 2020; 396: 461-462
        • Thombs BD
        • Kwakkenbos L
        • Carrier ME
        • et al.
        Protocol for a partially nested randomised controlled trial to evaluate the effectiveness of the scleroderma patient-centered intervention network COVID-19 home-isolation activities together (SPIN-CHAT) program to reduce anxiety among at-risk scleroderma patients.
        J Psychosom Res. 2020; 135110132
        • Kim SY
        • Flory J
        • Relton C
        Ethics and practice of trials within cohorts: an emerging pragmatic trial design.
        Clin Trials. 2018; 15: 9-16
      20. Kai Kupferschmidt. One U.K. trial is transforming COVID-19 treatment. Why haven't others delivered more results?. Sciencemag.org. July 2, 2020. Available at:https://www.sciencemag.org/news/2020/07/one-uk-trial-transforming-covid-19-treatment-why-haven-t-others-delivered-more-results. Accessed March 2, 2021.

      21. ECRIN - European Clinical Research Infrastructure Network. Fast track Procedure for COVID 19 Clinical Trials. Available at: https://ecrin.org/fast-track-procedure-covid-19-clinical-trials. Accessed February 26, 2021.

        • Mather N
        How we accelerated clinical trials in the age of coronavirus.
        Nature. 2020; 584: 326
      22. Swissethics. Swiss Association of Research Ethics Committees. Information on the Coronavirus: In Order to Promote Transparency and Facilitate Research Coordination, Swissethics Is Now Also Publishing Research Projects That Have Been Submitted But Not Yet Approved. Available at: https://swissethics.ch/en/news/2020/03/30/informationen-zum-corona-virus-um-die-transparenz-zu-foerdern-und-die-forschungskoordination-zu-erleichtern-veroeffentlicht-swissethics-neu-auch-eingereichte-aber-noch-nicht-bewilligte-forschungsprojekte. Accessed February 26, 2021.

        • DeMets DL
        • Fleming TR
        Achieving effective informed oversight by DMCs in COVID clinical trials.
        J Clin Epidemiol. 2020; 126: 167-171
        • Bompart F
        Ethical rationale for better coordination of clinical research on COVID-19.
        Research Ethics. 2020; 16: 1-10
      23. World Health Organization. “Solidarity” Clinical Trial for COVID-19 Treatments. Available at: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-coronavirus-2019-ncov/solidarity-trial-accelerating-a-safe-and-effective-covid-19-vaccine. Accessed February 26, 2021.

      24. University of Oxford. Oxford Leads First Trial Investigating Dosing With Alternating Vaccines. Available at: https://www.ox.ac.uk/news/2021-02-04-oxford-leads-first-trial-investigating-dosing-alternating-vaccines. Accessed February 26, 2021.

        • Angus DC
        • Gordon AC
        • Bauchner H
        Emerging lessons from COVID-19 for the US clinical research enterprise.
        JAMA. 2021; 325: 1159-1161