Stellungnahme des IQWiG zu 'ICH E9 (R1) Addendum über Schätzwerte und Sensitivitätsanalysen in klinischen Studien zum Leitfaden über statistischen Prinzipien für klinische Studien‘ (EMA/CHMP/ICH/436221/2017) (Englisch)

19. Februar 2018

Comments from:

Institute for Quality and Efficiency in Health Care (IQWiG)
Im Mediapark
50670 Cologne
Germany

Please note that these comments and the identity of the sender will be published unless a specificjustified objection is received. When completed, this form should be sent to the European Medicines Agency electronically, in Word format (not PDF).

1. General comments

IQWiG appreciates the opportunity to comment on the draft guideline.

The new ICH E9 (R1) addendum on estimands and sensitivity analysis represents a mixture of useful clarifications, trivial explanations (neglecting well-known approaches of evidence-based medicine), and a number of critical issues.

From a Health Technology Assessment (HTA) point of view, some of the listed estimands are not relevant and cannot be estimated in an unbiased manner. By suggesting these five different strategies without clarifying which of them would be a general or minimum requirement and which of them might only be used as sensitivity analyses or in special situations, the addendum weakens the requirement of robust analyses for regulatory decision-making.

In addition, for HTA it is essential that data collection for all endpoints is carried out for all patients up to the end of the study. We see the danger that, for example, the “while on treatment” strategy described in this addendum will be used as a justification for refraining from endpoint data collection after discontinuation of the initial treatment and will thus jeopardise the analyses required for HTA. In addition, this is not consistent with the recently adopted “Guideline on evaluation of anticancer medicinal products in man” (EMA/CHMP/205/95 Rev. 5, compare section “Extended safety data collection”).

We recommend revising the addendum, taking the well-known PICOS approach into account, and avoiding estimands that cannot be estimated without a high risk of bias and that contradict the statistical principles for clinical trials of the ICH E9 guideline.

IQWiG strongly supports a revision of the addendum, because only two of the described strategies (treatment policy, composite) should be used in general as the main analysis. The other three strategies (hypothetical, principal stratum, while on treatment) are useful only as a possible supplementary analysis for hypothesis generation or sensitivity analysis in special situations. So far, this does not become clear from the addendum. Therefore, there is the risk that the addendum is considered to be supportive of inappropriate analyses of clinical trial data.

2. Specific comments on text

Lines 119-123

Comment: It is trivial that a clear scientific question is required before parameters are estimated. The well-known PICOS approach (participants, interventions, comparators, outcomes, and study design) should be taken into account. The given series of items on the one hand goes beyond the PICOS approach (handling of intercurrent events and specification of the effect measure), but on the other hand is incomplete (intervention and comparator are missing).

Proposed change (if any): The given series of items should build on the well-known PICOS approach with appropriate additions.

Lines 151-157

Comment: In the given series of items A to D the important items “intervention” and “comparator” are missing.

Proposed change (if any): Please add the items “intervention” and “comparator” to the described items A to D.

Lines 210-121

Comment: It is incorrect that the treatment policy strategy "cannot be implemented when values for the variable after the intercurrent event do not exist for all subjects". For example, imputation techniques can be used to also include subjects with missing data after the intercurrent event.

Proposed change (if any): Please change the statement that the treatment policy strategy cannot be implemented to the statement that the treatment policy leads to problems when values for the variable after the intercurrent event do not exist for all subjects.

Lines 232-247

Comment: We question the validity and utility of the hypothetical strategy. Even if a valid parameter estimation could be performed in the hypothetical scenario that an observed intercurrent event had not happened, what is the value of this estimation in practice where intercurrent events are occurring?

Moreover, no methods are available to estimate estimands in hypothetical scenarios with a low risk of bias. Maybe there are situations where estimands for hypothetical scenarios make sense as additional information for hypothesis generation or sensitivity analysis. Therefore, the hypothetical strategy should not be described as an option for the main analysis.

Proposed change (if any): Please delete the hypothetical strategy from the available options for the main data analysis. Define the hypothetical strategy as a possible supplementary analysis for hypothesis generation or sensitivity analysis in special situations.

Lines 248-263

Comment: The "principle stratum strategy" is a purely hypothetical construct. Due to the given reason (confounding), principal strata could not be formed by subsets of patients without intercurrent events. Therefore, no methods are available to deal adequately with purely hypothetical principal strata.

Proposed change (if any): Please delete the principle stratum strategy from the available options for the main data analysis. Define the principle stratum strategy as a possible supplementary analysis for hypothesis generation or sensitivity analysis in special situations.

Lines 264-271

Comment: The restriction of the data analysis to the period of treatment continuation leads to serious problems due to different follow-up times. Therefore, this strategy should be avoided in general. Maybe there are situations where the “while on treatment” estimand makes sense as additional information for hypothesis generation or sensitivity analysis. However, the “while on treatment” strategy should not be described as an option for the main analysis.

Proposed change (if any): Please delete the “while on treatment” strategy from the available options for the main data analysis. Define the “while on treatment” strategy as a possible supplementary analysis for hypothesis generation or sensitivity analysis in special situations.

Lines 272-276

Comment: The five strategies are listed on the same level although only two strategies should be used as the main analysis in practice.

Proposed change (if any): Please divide the list of strategies into two parts. One part with options for the main analysis (treatment policy, composite) and a subordinate part with options for supplementary analyses in special situations (hypothetical, principal stratum, while on treatment).

Lines 302-303

Comment: The formulation "Some estimands, in particular those that are estimated using the observed data,…" is unclear and makes no sense.

If it means that an estimand is sometimes defined by the data observed, the statement is invalid because theoretical parameters should not be defined by the data observed. If it means that some estimands are estimated by the data observed and others not, the statement is of no use, because an estimand is only meaningful if it is estimable by means of the data observed.

Proposed change (if any): Please delete or revise the statement "Some estimands, in particular those that are estimated using the observed data,…".

Lines 337-338

Comment: The following statement is unclear "… but main and sensitivity estimators cannot be identified that are agreed to support a reliable estimate or robust inference."

Proposed change (if any): Please clarify what is meant by the statement "… but main and sensitivity estimators cannot be identified that are agreed to support a reliable estimate or robust inference."

Lines 464-465

Comment: It is correct that "Estimation for an estimand … will require stronger and untestable assumptions if measurements are not collected following intercurrent events." Therefore, every effort should be made to collect all relevant data after the occurrence of an intercurrent event.

Proposed change (if any): Please add the statement that every effort should be made to collect all relevant data after the occurrence of an intercurrent event.

Lines 468-471

Comment: It is correct that "… the estimation of estimands constructed using a strategy that requires a hypothetical scenario to address an intercurrent event entails careful specification of the hypothetical conditions and will necessarily rely on modelling assumptions that are untestable …". Therefore, the corresponding analysis should not be used as the main analysis for decision-making.

Proposed change (if any): Please add the statement that methods relying on strong untestable assumptions should not be used as the main analysis for decision-making.

Lines 472-473

Comment: It is correct that "… estimation of a treatment effect within a principal stratum of the population will be confounded unless the subjects within that stratum can be identified before randomisation." If the subjects can be identified before randomisation, the principal stratum strategy is nothing more than a conventional subgroup analysis. If this is not the case, the principal stratum strategy can only be used as a supplementary analysis but not as the main analysis for decision-making.

Proposed change (if any): Do not use the term "principal stratum strategy" for situations of a conventional subgroup analysis. In all other cases, do not describe the principal stratum strategy as an option for the main analysis.

Line 615

Comment: The method for statistical analysis is described as "… analysis of variance model with treatment group as a factor …". In the situation considered, the corresponding ANOVA model is reduced to the conventional t-test.

Proposed change (if any): Please replace "analysis of variance model" by "t-test".

Line 682

Comment: In the situation considered, the use of logistic regression is not required. A simple 2x2 table with an adequate statistical test would be sufficient.

Proposed change (if any): Please replace "logistic regression" by "2x2 table with an adequate statistical test".

Lines 692-713

Comment: We question the usefulness of a hypothetical setting in which it is assumed that rescue medication was not available. No regulatory decisions should be based upon such an analysis.

Proposed change (if any): Please clearly describe that an analysis in hypothetical settings may be used as a supplementary analysis in special situations.

Lines 724-725

Comment: It is not difficult to identify members of this hypothetical population in advance; it is, in general, impossible.

Proposed change (if any): Please describe that it is, in general, impossible to identify members of this hypothetical population in advance and that such an analysis should only be used as a supplementary analysis in special situations.

Line 735

Comment: It is correct that "An appropriate analysis needs to account for this confounding." However, no possible methods are described, not even in an exemplary way. Indeed, no method is available that guarantees to account for all known and unknown confounders.

Proposed change (if any): Please add that there is no robust method available in practice to deal with all known and unknown confounders and that the corresponding analysis should only be used as a supplementary analysis in special situations.

Line 743

Comment: The defined variable "average of the designated measurements while on randomised treatment" frequently leads to serious problems because the corresponding comparison is unfair due to different follow-up times.

Proposed change (if any): Please describe the problems of unfair comparisons due to different follow-up times and add that the corresponding analysis should only be used as a supplementary analysis in special situations.

Lines 748-750

Comment: There is almost always an interest in trial objectives that would require the collection of data after switching to rescue medication.

Proposed change (if any): Please revise the statement and state that, in general, the collection of data after switching to rescue medication is required.

Line 802

Comment: Again, the consideration of the hypothetical setting in which rescue medication would not be available is of no use in practice (see above).

Proposed change (if any): Please add a clear statement that the corresponding analysis should only be used as a supplementary analysis in special situations.

Line 813

Comment: There is almost always an interest in trial objectives that would require the collection of data after switching to rescue medication.

Proposed change (if any): Please revise the statement and state that, in general, the collection of data after switching to rescue medication is required.

Lines 842-845

Comment: The important items “intervention” and “comparator” are missing.

Proposed change (if any): Please add the items “intervention” and “comparator”.