Electronic database searches returned records after removal of duplicates, of which records were identified as potentially relevant based on the title and abstract scan. An additional 14 titles were identified as potentially relevant from scanning reference lists of included studies. Of the full-text articles reviewed, 60 articles were included in this review Fig.
Adapted from: Liberati et al. Research paradigms discussed by year of publication note: research paradigms are not mutually exclusive. Across the three groups there were 9 women and 4 men. The findings from our literature review and focus group interviews are discussed according to each of these paradigms.
While each research paradigm is discussed separately, they are not mutually exclusive.
Much of the discussion in the literature and amongst focus group participants concerning evaluative clinical research in rare diseases centered on problems associated with the inherent small numbers of patients available for study and adequate recruitment for conventional RCTs, long considered the gold standard explanatory design with a low risk of bias [ 5 ]. The first author to highlight the challenges associated with fewer participants for clinical studies was Haffner, who took the perspective of a regulatory agency responsible for reviewing the safety and efficacy of orphan medicines [ 30 , 31 ].
Haffner argued that orphan medicines should be as well-scrutinized as medicines for more common diseases but recognized that conventional RCTs are not always feasible due to small numbers [ 30 , 31 ].
- Rotator Cuff Tear: Pathogenesis, Evaluation and Treatment!
- Challenges and Opportunities In The Treatment Of Rare Diseases.
- Hematopathology: A Volume in the High Yield Pathology Series!
- See a Problem?.
- Challenges and Opportunities In The Treatment Of Rare Diseases!
- Account Options!
Some alternative research methods or design features for demonstrating safety and efficacy that may be acceptable to a regulatory agency were suggested, including the use of multicenter studies, crossover trials, randomized withdrawal trials, open label studies, open protocol studies, and incorporating historical controls or composite or surrogate endpoints [ 30 , 31 ]. The discussion concerning explanatory evidence generation for clinical interventions for rare diseases continued from these early publications to the present day Fig.
Others elaborated on the issues brought forth by Haffner and offered more suggestions to overcome the challenges related to small numbers and limited feasibility of conventional RCTs, while preserving internal validity and protecting against bias and confounding [ 2 , 4 , 18 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 ]. While participants in our focus groups highlighted the limited feasibility of conventional RCTs because of small sample sizes, there was little emphasis in the focus group discussions on specific strategies that might be used to overcome this challenge.
Thus, most of the results presented under the paradigm of explanatory evidence generation are derived from our meta-narrative literature review. In general, the research methods or study design features that have been proposed in the literature to address small numbers while retaining internal validity and thus an explanatory focus have concentrated on three overarching strategies: i enhancing statistical efficiency at the design phase, so that fewer participants are required to conduct a robust evaluation; ii using Bayesian rather than frequentist analysis methods, also to reduce the number of participants required; and iii making participation more appealing to patients and families by maximizing time spent on the active treatment.
Several methodological reviews were published on this topic in the last decade [ 36 , 39 , 40 , 42 , 45 , 46 , 61 ], some of which provided more detail about the methods described below; here we focus on the most commonly suggested research designs that focus on minimizing bias to maximize internal validity and explanatory power. Strategies that have been proposed for enhancing statistical efficiency at the design phase for clinical evaluative studies of rare disease treatments include factorial trials and adaptive designs.
Factorial trials are designed to test multiple treatments simultaneously using the same study population, thus reducing the overall number of participants needed [ 2 , 33 , 39 , 40 , 46 , 49 , 53 , 57 ]. However, authors have pointed out that this reduction in sample size only holds assuming there is no interaction between the treatments being administered concurrently; otherwise, statistical efficiency is lost [ 40 ].
Two commonly discussed adaptive trial strategies are response-adaptive randomization and group sequential design [ 36 , 40 , 46 , 53 , 59 , 61 ]. Group sequential designs do not have a predetermined sample size, rather, small groups of participants are recruited over several phases and data are analyzed at the end of each phase to assess safety, futility, efficacy, or a combination of these until enough data have been accrued to justify study termination [ 59 , 61 ]. Simulation studies have shown that sequential design approaches may, but do not always, reduce the eventual sample size compared to fixed sample size designs [ 35 , 53 , 62 ].
While adaptive trial strategies are often reported as a means to enhance statistical efficiency, some authors have questioned their usefulness based on the paucity of published practical application in the context of rare diseases [ 40 , 59 ].
Patient groups: our strongest weapon against rare diseases
For conventional RCTs with small sample sizes, achieving sufficient statistical power to detect differences in treatment effects, especially when the treatment effect is expected to be modest, is challenging [ 52 ]. Several authors have argued as early as that Bayesian techniques would be better suited in this context relative to standard frequentist approaches to analysis, because a Bayesian analysis is not as compromised by small numbers and offers more direct conclusions [ 32 , 34 , 41 , 44 , 45 , 48 , 50 , 51 , 52 ].
In such approaches, previously collected data or expert opinion is used to generate a prior probability posterior distribution for the unknown treatment effect, and Bayes theorem is applied as new data are accumulated to update the posterior distribution for the new treatment and inform clinical practice [ 48 , 52 ]. As an example, Johnson and colleagues reanalyzed data from an RCT of methotrexate versus placebo in 73 patients with scleroderma, and demonstrated that methotrexate had more favorable odds of being beneficial for patients when a Bayesian approach was applied compared to the non-statistically significant findings obtained through a frequentist approach [ 32 ].
- FDA Updates Draft Guidance on Rare Diseases: Some Key Takeaways You Need to Know.
- Atmospheric Science Across the Stratopause.
- Featured channels?
- Patient groups: our strongest weapon against rare diseases.
- Apostles Creed: and its Early Christian Context.
- Rare Diseases and Orphan Drugs : Jules J. Berman : ?
While several authors argued that Bayesian statistics offer an alternative approach to the analysis of small numbers of participants, some criticized the subjectivity in establishing prior distributions and were skeptical of the acceptance of results obtained using Bayesian statistics at the regulatory level [ 34 , 36 , 45 , 48 ]. Therefore, study designs that make participation more appealing by maximizing time spent on- or guaranteeing provision of- the active treatment have been suggested [ 4 , 33 , 36 , 38 , 39 , 40 , 41 , 42 , 44 , 45 , 46 , 47 , 49 , 51 , 56 , 57 , 60 ].
I mean, certainly there has been trials to try to do that. The randomized placebo-phase design has the same design features of a conventional RCT, except that the time from enrollment in the study to the start of the experimental treatment is randomized for all participants [ 56 ]. All participants eventually receive the experimental treatment, and effectiveness is determined based on whether a response is observed sooner among those that received the treatment earlier [ 56 ].
Similarly, randomized withdrawal, early escape, and stepped wedge trials reduce time spent in a control arm or ensure that all participants eventually receive the intervention being studied, and have been proposed as alternative approaches to evaluate clinical interventions for rare diseases [ 40 ]. Crossover trials and n-of-1 trials also guarantee that participants receive the active treatment, but are different than conventional RCTs in that the treatment sequence is randomized with a washout period in between treatment regimens, such that each participant acts as his or her own control [ 2 , 36 , 41 , 53 , 57 ].
As some authors reported, n-of-1 trials are often embedded in clinical practice to help healthcare providers determine the best treatments for their patients [ 2 , 36 , 57 ]. While several authors have examined the advantages of crossover and n-of-1 trials, others have discussed the risk of carryover and period effects between phases, and have argued that these designs are generally not suitable for diseases that have an unstable disease course or for interventions that are not fast-acting with reversible effects [ 2 , 18 , 33 , 36 , 39 , 44 , 46 , 53 ].
The three overarching strategies and associated research methods discussed above are not mutually exclusive, rather there is significant overlap among them in the literature. For example, in addition to being an attractive option for participants, crossover trials are also considered statistically efficient and reduce the number of participants needed because each participant acts as his or her own control [ 2 , 18 , 33 , 36 , 39 , 40 , 44 , 46 ].
Similarly, authors have stated that trials using adaptive randomization can be attractive to participants because the likelihood of being randomized to the less effective treatment arm is reduced over time [ 36 , 40 , 46 , 53 , 59 , 61 ]. Bayesian methods are also reported as a common design feature of adaptive trials as a means of improving statistical efficiency [ 34 , 42 , 59 ]. They have also been proposed as a means to combine results from multiple n-of-1 trials and enhance the usability of n-of-1 trial data in answering population-level questions about treatment efficacy and effectiveness [ 51 ].
A criticism of explanatory evidence generation reported both in the literature and in focus group discussions was that studies designed to evaluate the efficacy of an intervention typically limit enrolment to a very homogenous group of participants, which strengthens the robustness of the causal interpretation of the findings, but at the expense of a reduction in the external validity or generalizability of study results [ 4 , 18 , 44 , 60 ].
Because rare diseases typically exhibit substantial clinical heterogeneity discussed in the following section , some authors have questioned the suitability of the above-mentioned approaches for evaluating clinical interventions for rare diseases [ 4 , 18 , 44 , 60 ]. Additionally, authors have argued that many conventional RCTs and other explanatory studies are short in duration, often due to resource constraints, and do not allow for adequate assessment of long term treatment effects, further compromising external validity [ 4 , 18 , 57 ].
Finally, some authors were concerned that unfamiliar approaches to research design, such as adaptive randomization or n-of-1 trials would not be accepted by regulatory agencies and other policy decision-making bodies [ 36 ]. Partly in response to some of these concerns, other research paradigms for evaluating clinical interventions for rare diseases have evolved. It is well established that there is a high degree of clinical heterogeneity among rare disease patients, such that patients with the same specific disease might have drastically different clinical manifestations based on patient characteristics such as age, disease characteristics such as residual enzyme activity levels, or for unknown reasons, and may respond differently to a given intervention [ 18 , 42 ].
As several authors have discussed, this clinical heterogeneity is often not accounted for in conventional RCTs, and has raised concern among stakeholders about the applicability of study results to patients with clinical manifestations different from those included in RCTs [ 4 , 18 , 44 , 60 ]. And this is one of the big problems, like [name] mentioned, how do we apply this clinically to a larger population of these patients?
Are the results, for instance, with infantile-Pompe, how do we relate that to an adult Pompe patient?
It makes it very difficult for us to know where and when these therapies are going to work. In response to concerns about the external validity of study results, several authors and focus group participants have advocated for study designs that may compromise internal validity to some extent, by shifting away from the explanatory RCT, in order to address real-world effectiveness [ 2 , 4 , 7 , 18 , 42 , 44 , 45 , 46 , 47 , 55 , 57 , 58 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 ].
This effect to observational studies and looking at outcome differences in naturally, sort of, selected difference maybe as helpful in rare diseases I think as the designed studies. Wilcken suggested that for some rare diseases, conventional RCTs remained possible, but for others, observational studies with historical controls could be used to evaluate treatment effectiveness [ 7 ]. Since that initial publication, many authors have discussed research designs that take a more pragmatic approach to evaluating treatment effectiveness in rare diseases, and often explicitly attempt to include a broader patient population and longer-term observation in natural settings.
These designs include: pragmatic clinical trials, observational studies e. While participants in our focus groups questioned the suitability of explanatory RCTs for establishing effectiveness of clinical interventions for rare diseases, little of the discussion focused on specific solutions to overcome this challenge. Incorporating more pragmatic features into RCTs has been suggested as a means to improve external validity while maintaining the element of randomization to help control for unmeasured confounding and maintaining other standard methodological features of explanatory RCTs, such as blinded outcome assessments [ 18 , 45 , 57 ].
These pragmatic RCTs feature design elements that better reflect actual clinical practice, including: enrolling participants with differing clinical presentations, taking into consideration the system of care in which the new treatment will be delivered e. Authors have criticized pragmatic RCTs because they do still estimate average treatment effects and thus are not necessarily better suited to investigating potential heterogeneity of treatment effects relative to explanatory RCTs [ 18 ]. Among the most common observational rare disease research designs discussed in the studies we reviewed are patient registries [ 4 , 18 , 42 , 47 , 58 , 64 , 65 , 67 , 72 , 73 , 74 , 77 , 80 ] and cohort studies [ 68 , 78 ].
FDA Updates Draft Guidance on Rare Diseases: Some Key Takeaways You Need to Know
We identified several examples of registries being used to evaluate treatment effectiveness of interventions for rare diseases, for example, enzyme replacement therapy for lysosomal storage disorders [ 72 ]. The International Collaborative Gaucher Group Registry was established in and, at the time of the publication of a paper by Jones and colleagues , had collected longitudinal clinical data for almost patients [ 72 ].
Several authors stated that an additional advantage of registries is that they can be used to identify potential participants for recruitment into future research studies, including clinical trials [ 18 , 67 , 73 , 76 , 77 ]. Some authors have also suggested that observational patient registries may play an important role in post-market evaluation of interventions for rare diseases by serving as a platform to collect longitudinal clinical and quality of life data [ 47 ]. While observational patient registries are an attractive method for the evaluation of longer term outcomes in real-world settings, some authors reported that results remain prone to residual confounding in the absence of randomization, especially confounding by indication when patient characteristics influencing the choice of treatment also influence the outcome [ 18 , 44 ].
A few authors discussed variability in the quality of registry data, as observational patient registries tend to be heterogeneous in the depth of data collection and the definitions applied to included data elements, particularly in the context of the multi-center and sometimes multi-national nature of rare disease research [ 42 , 65 ]. In addition, some authors described the potentially important influence of complete case ascertainment and data collection on the accuracy of study results, particularly given that registry participation may be associated with receipt of particular treatments or lead to different investigations [ 67 , 73 , 81 ].
The key feature of the clinically-integrated randomized trial is that there is no difference between the care a patient routinely receives, follow-up, payment, or documentation e. In the context of rare diseases, the authors argued that the clinically-integrated randomized trial is attractive because there is often considerable uncertainty about the most effective course of treatment for patients and that trials could easily be conducted worldwide to maximize the number of participants [ 63 ].
The cmRCT seeks to enroll an observational cohort of patients, with participants routinely reporting on a minimum set of core outcomes [ 75 , 82 ]. At the time of enrollment in the cohort, participants give their consent for 1 their longitudinal data to be used in aggregate; and 2 to be randomly selected to participate in potential RCTs of new or existing interventions with the understanding that only those who have been selected to be offered the intervention under study will be contacted [ 75 , 82 ].
Those who are eligible for the RCT, but who were not randomly selected to be offered the intervention serve as the control group and are not contacted about the study [ 75 , 82 ]. Finally, there is discussion in this literature about other observational designs such as case-control studies, small case series and case reports; however, these approaches are not commonly suggested as potential solutions for improving pragmatic evidence generation for establishing effectiveness of treatments for rare diseases. Some authors have suggested that case-control designs, where individuals who have experienced a certain outcome cases are matched to and compared with individuals who have not experienced the outcome of interest controls , are well suited for studying rare diseases, particularly in instances where there could be a long lag time between the treatment and outcome of interest [ 2 , 80 ].
However, there are concerns about the potential for introducing selection bias in choosing controls [ 2 ]. Other authors have argued the importance of case series and case reports in the context of establishing treatment effectiveness for rare diseases [ 47 , 66 ]. Case series and case reports typically include in-depth information related to clinical manifestations of disease, treatment, and follow-up for a single patient or small group of patients [ 47 , 66 ]. Similar to the concept of using case reports as pragmatic evidence, several focus group participants reported relying on some anecdotal evidence to help inform medical-decision making:.
Because we deal with very rare disorders sometimes, and you often go to clinicians who have seen these conditions and have treated them, and may take their point of view about a certain treatment.
Rare Diseases and Orphan Drugs
So, I think all of the studies and designs, including anecdotal evidence, I personally use that in determining whether I think about a treatment for a patient. So, I like to have the bad and the good ones too, and then make my mind and take better decisions. As previously discussed, some authors have suggested incorporating pragmatic elements into RCTs [ 18 , 45 , 57 ], while others have proposed methods to overcome challenges in non-randomized studies. For example, Cole and colleagues demonstrated the use of case-control matching using the risk-set method for participants enrolled in the International Collaborative Gaucher Group Registry [ 69 ].
Use of propensity scores to match participants has also been suggested as a means of reducing the risk of bias in observational studies of rare diseases [ 44 ]. One of the main criticisms, both in the literature and by focus group participants, of highly internally valid, explanatory study designs is their tendency to rely on short-term, and often surrogate, outcomes that are not necessarily clinically meaningful [ 9 ].
Rather than what the clinician may feel for a particular rare disease is far more important. Only in the last decade Fig. This discussion emphasizes the need for outcomes that are of direct importance to patients and caregivers. Connected to the paradigm of explanatory evidence generation, some authors have suggested the use of surrogate outcomes as proxies for patient-oriented outcomes such as survival or quality of life because they can be measured relatively quickly and require fewer participants to reach statistical efficiency [ 33 , 83 , 84 , 85 ].
For example, in , Kinder and colleagues reported that functional outcomes such as exercise tolerance, survival, and quality of life were the most salient outcomes to consider for rare lung disease studies because they have undeniable meaning for patients; however, the authors also described the limited feasibility of conducting explanatory RCTs that include these outcomes and argued that surrogate outcomes could therefore be developed and used as proxies for patient-oriented outcomes [ 33 ].