Stellungnahme zum "Reflexionspapier zum Nachweis der Wirksamkeit auf der Grundlage von einarmigen Studien als pivotaler Evidenz für eine Zulassung" (Englisch)
IQWiG appreciates the opportunity to provide comments on the reflectionvpaper.
We agree with the general statements that SATs lack fundamental features that are required to avoid bias and that therefore in general SATs do not allow a causal interpretation of estimated effects. We also agree that in exceptional cases where there is no doubt of the outcome in the absence of an active treatment the treatment effect can be reliably estimated from the observed outcome in a SAT.
However, we miss a clear conclusion that SATs should in general be avoided and that this design is only an option in rare exceptional cases.
We support the view that the considerations regarding SATs also refer to non-randomised studies and any other situation in which samples of patients are compared without randomisation of the treatment arms. However, we miss the clear statement that effect estimations in situations without randomisation of the compared treatment arms require individual patient data (IPD). Even if guidance on comparisons with external data is beyond the scope of the reflection paper, it should at least be added that such comparisons require access to full IPD information in order to adjust for confounding.
In many cases, SATs mean that external controls are needed to describe treatment effects. Therefore, as stated in the reflection paper, effect estimates from SATs can be compared to non-randomised studies. While the paper describes a number of difficulties arising from this situation, it does not make a clear recommendation regarding external
controls.
In contrast, the FDA guidance “Considerations for the Design and Conduct of Externally Controlled Trials for Drugs and Biological Products” (February 2023) provides clearer guidance in stating: “In many situations, the likelihood of credibly demonstrating the effectiveness of a drug of interest with an external control is low, and sponsors should choose a more suitable design, regardless of the prevalence of the disease.” FDA is also clearly stating example scenarios which are generally not suitable for externally
controlled trials (when the natural history of the disease of interest is not understood sufficiently or when the disease course is considered wellunderstood but is variable). We suggest adding similar considerations and recommendations to the reflection paper.
There may be rare cases where SATs can demonstrate drug activity (without reference to external controls) to an extent considered sufficient by regulatory authorities. As described in the reflection paper, these cases are limited to specific situations and a very limited choice of endpoints. Regardless of this, post-approval decisions about the actual use of a new medicine in a healthcare system require comparison with available
treatment options (including non-drug interventions or best supportive care). These decisions need to be made immediately after approval. Therefore, the use of SATs for regulatory approval cannot be considered in isolation. Regulatory approval based on SATs without taking into account the post-authorisation decisions will lead to delays in patient access to new medicines in many cases. Rather, it is necessary to ensure the availability of comparative data on relevant endpoints at the time of marketing authorisation. Ideally, this data will be from randomised controlled trials. In exceptional cases, externally controlled comparisons using appropriate methodology may be sufficient. If regulators such as the EMA accept SATs for their decision-making, they should at least make sponsors aware of the need for parallel comparative data generation for HTA. This also raises the question of whether conducting appropriately controlled trials in the first place is not the better strategy for accelerating patient access to evidence-based new treatments. In summary, the goal should not be accelerated market access per se, but accelerated evidence-based market access for the benefit of current and future patients.
Comment:
We fully agree that in general the interpretation of estimated effects based on SATs must rely on assumptions and are therefore less reliable compared to effect estimates from adequate RCTs. It should be added that the consequence isthat SATs in general do not represent a proper basis for drug approval.
Proposed change (if any):
Please add a statement like this: “As a consequence, the derived magnitude of effects is more difficult to interpret, and less reliable. Therefore, with rare exceptions, the likelihood of credibly demonstrating the efficacy (and safety) of a drug of interest with a SAT is low, and sponsors should choose a more suitable design.”
Comment:
We agree that in situations where individual outcomes in a SAT for the defined endpoint within the designated follow-up could not have occurred without active treatment in any patient who entered the trial, the SAT is able to isolate the treatment effect on that specific endpoint. We also agree that this situation allows a causal interpretation of the effect of the treatment.
However, it should be added that this situation is a very rare exception and does not justify the use of a SAT design in other situations.
Proposed change (if any):
Please add a statement like this: “Conceptually, this can allow a causal interpretation of the effect of the treatment, despite the limitations in study design. However, this situation is a very rare exception anddoes not justify the use of a SAT design in other situations.”
Comment:
It should be added that contrasts to external data require access to the full IPD information to apply adequate methods for confounder adjustment. It should also be added that a systematic approach is required to identify all relevant confounders and that all relevant confounders should be taken into account in the data analysis.
Proposed change (if any):
Please add statements like this: “In other cases, treatment effect estimates defined for SATs may include contrasts to external control group data. This approach requires access to the full individual patient data (IPD) information to apply adequate methods for confounder adjustment. A systematic approach to identify all relevant confounders is required and all relevant confounders have to be taken into account in the data analysis.”
Comment:
We fully agree that the absence of the randomised control arm substantially increases the risk of bias and thus reduces internal validity. It should be added that the consequence isthat SATs in general do not represent a proper basis for drug approval.
Proposed change (if any):
Please add a statement like this: “The absence of the randomised control arm substantially increases the risk of bias and thus reduces internal validity. As a consequence, with rare exceptions, the likelihood of credibly demonstrating the efficacy (and safety) of a drug of interest with a SAT is low, and sponsors should choose a more suitable design.”
Comment:
The statement that the variability of individual outcomes for the experimental arm is directly observed, but not for the hypothetical control is not correct if there is access to the IPD information in the control arm from, e.g., another SAT. It should be added that the possibility to observe variability also in the control arm is another reason that access to full IPD information is required if contrasts to external control group data are considered.
Proposed change (if any):
Please include the situation that full IPD information is available for external control group data.
Comment:
We agree that the potential impact of unknown prognostic or predictive variables cannot be controlled in SATs and that in practice, the estimation of or control for the impact of known prognostic variables is not always feasible. We also agree that it is not possible to disentangle prognostic from predictive effects based on the results derived from SATs.
Nevertheless, it should be added that a systematic approach is required to identify relevant prognostic or predictive variables and all relevant prognostic or predictive variables have to be taken into account.
Proposed change (if any):
Please add a statement like this: “In particular, it is not possible to disentangle prognostic from predictive effects based on results derived from SATs. Nevertheless, a systematic approach is required to identify relevant prognostic or predictive variables and all relevant prognostic or predictive variables have to be taken into account in the data analysis. If this is not possible, a SAT design is not a valid option in the
framework of drug approval.”
The text refers to a comparison of the SAT against external clinical data as a “direct comparison”. This is unfortunate because it interferes with the terminology of direct and
indirect comparisons used for statistical analyses (please see the EUnetHTA21 Methodological Guidance on direct and indirect comparisons for a suggested terminology: https://www.eunethta.eu/d4-3/).
Proposed change (if any):
“In exceptional cases, the assessment of efficacy is envisaged to be informed by a comparison of the SAT against external clinical data (i.e. an external control). … While methods that incorporate external data into the analysis come with a promise to provide useful insights and potentially reduce bias, they add complexity to prespecification and rely on additional assumptions that are often not transparent. Consequently, approaches that incorporate external data should be carefully evaluated on a case-by-case basis.
Comment:
Even if guidance on comparisons with external data is beyond the scope of the reflection paper, it should be added that for comparisons with external data access to full IPD information and a thorough statistical analysis plan (SAP) are required, which describes in detail all planned analyses for confounder adjustment.
Proposed change (if any):
Please add a statement like this: “Consequently, approaches that directly incorporate external data should be carefully evaluated on a case-by-case basis. For this option access to full IPD information and a thorough statistical analysis plan (SAP) are required, which describes in detail all planned analyses for confounder adjustment.”
Comment:
Even if the external control group matches well the enrolled trial population it cannot be expected that all relevant confounders are balanced. It should be added that it is required to use a systematic approach to identify all relevant confounders, to have access to full IPD information and take all relevant confounders into account in the data analysis.
Proposed change (if any):
Please add a statement like this: “Precisely pre-specify inclusion and exclusion criteria such that the enrolled trial population matches well the external information that assumptions are based on. Use a systematic approach to identify all relevant confounders, make certain that access to full IPD information is available and take all relevant confounders into account in the data analysis.”