Update assurance of a clinical trial after not stopping at an interim analysis: continuous endpoint

Author
Affiliation

Kaspar Rufibach

Methodology, Collaboration, and Outreach Group, PD DSS, Roche Basel

Published

Invalid Date

Terminology

In what follows when we talk about “effect” we always refer to actual effect, i.e. not standardized.

Setup

We assume that DDCP at the design stage has already been computed, as described in the corresponding tutorial.

The question we are answering here is now how the two DDCP values (one for each prior), 0.72 and 0.61, change during the trial when one of the following events happens:

  1. External information on the treatment effect becomes available.
  2. An interim analysis for futility and/or efficacy for the trial of interest itself is performed, and the trial is not stopped at that interim analysis. The question then is how the knowledge of not stopping at the interim analysis changes the Bayesian predictive probability. Computations are possible for both scenarios, when the interim effect estimate is known or unknown to the sponsor.

External information becomes available

Assume an early phase trial on a related molecule with 45 patients has read out. How should we update DDCP? The trial showed the following results:

# effect estimate and SE
mean1 <- 9.7
n_ext <- 45
sd_ext <- 32.4
se_ext <- sqrt(sd_ext ^ 2 / n_ext)

As discussed in Section 2.4 of Rufibach et al. (2016) the proposal is to simply update our prior distribution \(q\) with this data. If we have more than one external trial result we can either do this updating sequentially or by first meta-analyzing the multiple trials.

# ----------------------------------
# Normal prior:
# ----------------------------------
up1 <- NormalNormalPosterior(datamean = mean1, sigma = se_ext, n = 1, nu = priormean, 
                             tau = sd0)
bpp1 <- bpp_continuous(prior = "normal", successmean = mdd, stDev = stDev, 
                       n1 = n1, n2 = n2, priormean = up1$postmean, priorsigma = up1$postsigma)

# Normal prior
up1
$postmean
[1] 10.33355

$postsigma
[1] 4.200428
bpp1
[1] 0.726636

We find that updating the prior with the information from the external trial changes our Normal prior with mean = 12.30 and standard deviation 8.51 to a posterior with mean = 10.33 and increases DDCP from the initial 0.72 to 0.73. So why an increase if the posterior mean actually quantifies a lower effect compared to the prior? That is true, but we also have more observations for this effect, i.e. the posterior is more narrow, and the posterior mean is still better than the MDD.

# ----------------------------------
# pessimistic prior
# ----------------------------------
finalSE <- sqrt(stDev ^ 2 / n1 + stDev ^ 2 / n2)
bpp1_1 <- 1 - integrate(FlatNormalPosterior, lower = Inf, upper = Inf, successmean = mdd, 
                     finalSE = finalSE, interimmean = mean1, interimSE = se_ext, 
                     priormean = priormean, width = width1, height = height1)$value

# flat prior
bpp1_1
[1] 0.6729337

We find that updating the prior with the information from the external trial increases also this DDCP from the initial 0.61 to 0.67 if we use the pessimistic prior.

Update the prior distribution after not stopping at an interim analysis

Now, the next milestone in our trial of interest is an interim analysis for both, futility and efficacy:

# ---------------------------------------------------
# specifications for the interim analysis
# ---------------------------------------------------

# local significance levels based on rpact output
alphas <- 2 * design$stageLevels
alphas
[1] 0.003050646 0.048999543
# number of patients at interim analysis
n1_int <- ceiling(as.vector(sampleSize$numberOfSubjects1)[1])
n2_int <- ceiling(as.vector(sampleSize$numberOfSubjects2)[1])
c(n1_int, n2_int)
[1] 46 46
# MDDs at interim and final based on rpact output
mdd <- as.vector(sampleSize$criticalValuesEffectScaleUpper)
mdd
[1] 15.262428  7.023506
# efficacy boundary --> MDD at interim analysis
effi <- mdd[1]

# futility boundary --> chosen informally
futi <- 0

The interim analysis is planned after 92 patients (both arms combined), using local significance levels of \(\alpha_{inte} \ = \ 0.003\) for the interim and \(\alpha_{fin} \ = \ 0.049\) for the final analysis. From these local significance levels, together with the number of events at each analysis, the minimal detectable differences (the mean differences corresponding to the critical values of the hypothesis test, MDD) can be derived. These are 15.3 and 7.0.

In addition to the interim efficacy analysis, assume that a futility analysis was pre-specified in the iDMC charter, with interim futility boundary of \(\theta_{fut} \ = \ 0\). Thus, if the trial is not stopped at the interim analysis, we may assume for the mean difference estimate at the interim analysis that \({\hat \theta}_{inte} \in [\theta_{fut}, \theta_{eff}] \ = \ [0, 15]\).

Now we are interested how DDCP changes if we do not stop at that interim analysis. For illustrative purposes, we assess various scenarios: the interim is set up with futility and efficacy boundary, only one of the two, and we also provide updates using conditional power for the two extreme cases, namely that the estimate at the interim was known to be either equal to the futility or the efficacy boundary. Let us start with the Normal prior.

# ----------------------------------
# compute bpp after not stopping at interim, for Normal prior and various 
# assumptions on the amount of information we learn at the interim
#
# Note that if we are only interested on a "blinded" or "interval" update we can set
# IntFix to an arbitrary value. Below, this is the case whenever we set IntFix = 1
# ----------------------------------

# assuming we do not stop at a blinded interim analysis for futility and efficacy:
bpp3.tmp <- bpp_1interim_continuous(prior = "normal", successmean = mdd[2], stDev = stDev, 
                                    n1 = c(n1_int, n1), n2 = c(n2_int, n2), 
                                    IntEffBoundary = effi, IntFutBoundary = futi, IntFix = 1, 
                                    priormean = up1$postmean, propA = 0.5, thetas = thetas, 
                                    priorsigma = up1$postsigma)
bpp3 <- bpp3.tmp$"BPP after not stopping at interim interval"
post3 <- bpp3.tmp$"posterior density interval"

# assuming we do not stop at a blinded interim analysis for efficacy only:
bpp3_effi_only <- bpp_1interim_continuous(prior = "normal", successmean = mdd[2], stDev = stDev, 
                                 n1 = c(n1_int, n1), n2 = c(n2_int, n2), 
                                 IntEffBoundary = effi, IntFutBoundary = -Inf, IntFix = 1, 
                                 priormean = up1$postmean, propA = 0.5, thetas = thetas, priorsigma = 
                                 up1$postsigma)$"BPP after not stopping at interim interval"

# assuming we do not stop at a blinded interim analysis for futility only:
bpp3_futi_only <- bpp_1interim_continuous(prior = "normal", successmean = mdd[2], stDev = stDev, 
                                 n1 = c(n1_int, n1), n2 = c(n2_int, n2), 
                                 IntEffBoundary = Inf, IntFutBoundary = futi, IntFix = 1, 
                                 priormean = up1$postmean, propA = 0.5, thetas = thetas, priorsigma = 
                                 up1$postsigma)$"BPP after not stopping at interim interval"

# assuming we do not stop at an unblinded interim where we observe exactly the efficacy boundary
# (IntFix = effi): 
bpp4.tmp <- bpp_1interim_continuous(prior = "normal", successmean = mdd[2], stDev = stDev, 
                                    n1 = c(n1_int, n1), n2 = c(n2_int, n2), 
                                    IntEffBoundary = Inf, IntFutBoundary = -Inf, IntFix = effi, 
                                    priormean = up1$postmean, propA = 0.5, thetas = thetas, 
                                    priorsigma = up1$postsigma)
bpp4 <- bpp4.tmp$"BPP after not stopping at interim exact"[2, 1]
post4 <- bpp4.tmp$"posterior density exact"[, 1]

# assuming we do not stop at an unblinded interim where we observe exactly the futility boundary
# (IntFix = futi): 
bpp5.tmp <- bpp_1interim_continuous(prior = "normal", successmean = mdd[2], stDev = stDev, 
                                    n1 = c(n1_int, n1), n2 = c(n2_int, n2), 
                                    IntEffBoundary = Inf, IntFutBoundary = -Inf, IntFix = futi, 
                                    priormean = up1$postmean, propA = 0.5, thetas = thetas, 
                                    priorsigma = up1$postsigma)
bpp5 <- bpp5.tmp$"BPP after not stopping at interim exact"[2, 1]
post5 <- bpp5.tmp$"posterior density exact"     # same as post4[, 2]

We can also reproduce these computations using the probSuccess package discussed in the corresponding tutorial:

library(probSuccess)

# probSuccess:
# - make sure you specify the arguments correctly
# - note that probSuccess does not allow for non-1:1 randomization
# - computes the probability to be above the threshold (= successmean), 
# so we need to take 1 -:
PTSafterIA(TPP = mdd[2], type = "interval", IntIA = c(futi, effi), 
               sdIA = stDev, nIA = n1_int, nFinal = n1, prior = dnorm, 
               mean = up1$postmean, sd = up1$postsigma)
[1] 0.6963846
# this is the same as:
bpp3
[1] 0.6963846

And the same for the pessimistic prior:

# ----------------------------------
# compute bpp after not stopping at interim, for pessimistic prior and various 
# assumptions on the amount of information we learn at the interim
# here we do not assume an update with external studies, i.e. we use the
# initial prior
# ----------------------------------

# assuming we do not stop at a blinded interim analysis for futility and efficacy:
bpp3.tmp_1 <- bpp_1interim_continuous(prior = "flat", successmean = mdd[2], stDev = stDev, 
                                   n1 = c(n1_int, n1), n2 = c(n2_int, n2), IntEffBoundary = effi, 
                                   IntFutBoundary = futi, IntFix = 1, 
                                   priormean = priormeanflat, propA = 0.5, thetas = thetas, 
                                   width = width1, height = height1)
bpp3_1 <- bpp3.tmp_1$"BPP after not stopping at interim interval"
post3_1 <- bpp3.tmp_1$"posterior density interval"

# assuming we do not stop at a blinded interim analysis for efficacy only:
bpp3_1_effi_only <- bpp_1interim_continuous(prior = "flat", successmean = mdd[2], stDev = stDev, 
                                   n1 = c(n1_int, n1), n2 = c(n2_int, n2), IntEffBoundary = effi, 
                                   IntFutBoundary = -Inf, IntFix = 1, priormean = priormeanflat, 
                                   propA = 0.5, thetas = thetas, width = width1, 
                                   height = height1)$"BPP after not stopping at interim interval"

# assuming we do not stop at a blinded interim analysis for futility only:
bpp3_1_futi_only <- bpp_1interim_continuous(prior = "flat", successmean = mdd[2], stDev = stDev, 
                                   n1 = c(n1_int, n1), n2 = c(n2_int, n2), IntEffBoundary = Inf, 
                                   IntFutBoundary = futi, IntFix = 1, priormean = priormeanflat, 
                                   propA = 0.5, thetas = thetas, width = width1, 
                                   height = height1)$"BPP after not stopping at interim interval"

# assuming we do not stop at an unblinded interim where we observe exactly the efficacy boundary
# (IntFix = effi): 
bpp4_1.tmp <- bpp_1interim_continuous(prior = "flat", successmean = mdd[2], stDev = stDev, 
                                   n1 = c(n1_int, n1), n2 = c(n2_int, n2), IntEffBoundary = Inf, 
                                   IntFutBoundary = -Inf, IntFix = effi, priormean = priormeanflat, 
                                   propA = 0.5, thetas = thetas, width = width1, 
                                   height = height1)
bpp4_1 <- bpp4_1.tmp$"BPP after not stopping at interim exact"[2, 1]
post4_1 <- bpp4_1.tmp$"posterior density exact"

# assuming we do not stop at an unblinded interim where we observe exactly the futility boundary
# (IntFix = futi): 
bpp5_1.tmp <- bpp_1interim_continuous(prior = "flat", successmean = mdd[2], stDev = stDev, 
                                   n1 = c(n1_int, n1), n2 = c(n2_int, n2), IntEffBoundary = Inf, 
                                   IntFutBoundary = -Inf, IntFix = futi, priormean = priormeanflat, 
                                   propA = 0.5, thetas = thetas, width = width1, 
                                   height = height1)
bpp5_1 <- bpp5_1.tmp$"BPP after not stopping at interim exact"[2, 1]
post5_1 <- bpp5_1.tmp$"posterior density exact"

probSuccess does not implement the same prior as the pessimistic above, but also has a feature for a non-informative prior:

# using probSuccess:
PTSafterIA(TPP = mdd[2], type = "interval", IntIA = c(futi, effi), 
               sdIA = stDev, nIA = n1_int, nFinal = n1)
[1] 0.5385492
# this is virtually  the same as:
bpp3_1
[1] 0.5433146

So the two DDCPs, both based on a non-informal prior (which is not the identical one in both cases though!), are virtually identical.

Posterior densities

The resulting posteriors after the update with the external trial is depicted below. We also add to the figures the posterior after not stopping at the interim analysis that considers the efficacy and futility boundary, i.e. we may assume that \[{\hat \theta}_{inte} \in [\theta_{fut}, \theta_{eff}] \ = \ [0, 15].\]

These figures adapt Figure 1 in Rufibach et al. (2016) to our numbers.

par(las = 1, mar = c(4.5, 4.5, 4.5, 1), mfrow = c(1, 2), cex = 0.8)
leg <- c("prior", "posterior after external trial", 
         "posterior after external trial and\nnot stopping at interim")

# ----------------------------------
# Normal prior:
# ----------------------------------
xli <- priormean + 30 * c(-1, 1)
yli <- c(0, 0.12)
plot(0, 0, type = "n", xlim = xli, ylim = yli, xlab = "mean difference", ylab = "density", 
     main = "Normal prior density for\nmean difference")
abline(h = 0, lty = 3)
lines(thetas, dnorm(thetas, mean = priormean, sd = sd0), col = 2, lwd = 2)
lines(thetas, dnorm(thetas, mean = up1$postmean, sd = up1$postsigma), col = 3, lwd = 2)
lines(thetas, post3, col = 1, lwd = 2)
legend("topleft", leg, lty = 1, col = c(2:3, 1), bty = "n", lwd = 2)

# ----------------------------------
# pessimistic prior:
# ----------------------------------

# first we have to compute the posteriors after the external updates:
flatpost1 <- rep(NA, length(thetas))
for (i in 1:length(thetas)){
  flatpost1[i] <- estimate_posterior(x = thetas[i], prior = "flat", interimmean = mean1, 
                                     interimSE = se_ext, priormean = priormeanflat, width = width1, 
                                     height = height1)
}

plot(0, 0, type = "n", xlim = xli, ylim = yli, xlab = "mean difference", ylab = "density", 
     main = "Pessimistic prior density for\nmean difference")
abline(h = 0, lty = 3)
lines(thetas, dUniformNormalTails(thetas, mu = priormeanflat, width = width1, height = height1), 
      lwd = 2, col = 2)
lines(thetas, flatpost1, col = 3, lwd = 2)
lines(thetas, post3_1, col = 1, lwd = 2)

legend("topleft", leg, lty = 1, col = c(2:3, 1), bty = "n", lwd = 2)

Next, we also provide the posteriors after updating with the interim information assuming the effect estimate at the interim came to lie on one of the interim boundaries:

par(las = 1, mar = c(4.5, 4.5, 4.5, 1), mfrow = c(1, 2), cex = 0.8)

# ----------------------------------
# Normal prior:
# ----------------------------------
plot(0, 0, type = "n", xlim = xli, ylim = yli, xlab = "mean difference", ylab = "density", 
     main = "Normal prior density for\nmean difference")
abline(h = 0, lty = 3)
lines(thetas, post3, col = 1, lwd = 2)
lines(thetas, post4, col = 2, lwd = 2)
lines(thetas, post5, col = 3, lwd = 2)
leg2 <- c("interval knowledge", expression(hat(theta)*" = efficacy boundary"), 
          expression(hat(theta)*" = futility boundary"))
legend(-1, 3.5, leg2, lty = 1, col = 1:3, lwd = 2, bty = "n", 
       title = "posterior after not stopping at interim,")

# ----------------------------------
# pessimistic prior:
# ----------------------------------
plot(0, 0, type = "n", xlim = xli, ylim = yli, xlab = "mean difference", ylab = "density", 
     main = "Pessimistic prior density for\nmean difference")
abline(h = 0, lty = 3)
lines(thetas, post3_1, col = 1, lwd = 2)
lines(thetas, post4_1, col = 2, lwd = 2)
lines(thetas, post5_1, col = 3, lwd = 2)
legend(-1, 3.5, leg2, lty = 1, col = 1:3, lwd = 2, bty = "n", 
       title = "posterior after not stopping at interim,")

Table 1 in Rufibach et al. (2016)

Finally, we collect all the results computed above in one table, adapting Table 1 in Rufibach et al. (2016).

mat <- matrix(NA, ncol = 2, nrow = 9)
mat[, 1] <- c(pnorm2, pnorm1, bpp0, bpp1, bpp3, bpp3_futi_only, bpp3_effi_only, bpp4, bpp5)
mat[, 2] <- c(flat2, flat1, bpp0_1, bpp1_1, bpp3_1, bpp3_1_futi_only, bpp3_1_effi_only, 
              bpp4_1, bpp5_1)
colnames(mat) <- c("Normal prior", "Flat prior")
rownames(mat) <- c(paste("Probability for mean difference to be $\\le$ ", lims[1], sep = ""), 
paste("Probability for mean difference to be $\\ge$ ", lims[2], sep = ""), 
"DDCP based on prior distribution", "DDCP after external trial", 
"DDCP after external trial and not stopping at interim, assuming $\\hat \\theta \\in [\\theta_{fut}, \\theta_{eff}]$", 
"DDCP after external trial and not stopping at interim, assuming $\\hat \\theta \\in [\\theta_{fut}, \\infty]$", 
"DDCP after external trial and not stopping at interim, assuming $\\hat \\theta \\in [-\\infty, \\theta_{eff}]$", 
"DDCP after external trial and not stopping at interim, assuming $\\hat \\theta = \\theta_{eff}$", 
"DDCP after external trial and not stopping at interim, assuming $\\hat \\theta = \\theta_{fut}$")
mat <- as.data.frame(format(mat, digits = 2))
knitr::kable(mat)
Normal prior Flat prior
Probability for mean difference to be \(\le\) 0 0.607 0.546
Probability for mean difference to be \(\ge\) 10 0.074 0.254
DDCP based on prior distribution 0.717 0.606
DDCP after external trial 0.727 0.673
DDCP after external trial and not stopping at interim, assuming \(\hat \theta \in [\theta_{fut}, \theta_{eff}]\) 0.696 0.543
DDCP after external trial and not stopping at interim, assuming \(\hat \theta \in [\theta_{fut}, \infty]\) 0.768 0.814
DDCP after external trial and not stopping at interim, assuming \(\hat \theta \in [-\infty, \theta_{eff}]\) 0.648 0.295
DDCP after external trial and not stopping at interim, assuming \(\hat \theta = \theta_{eff}\) 0.989 0.990
DDCP after external trial and not stopping at interim, assuming \(\hat \theta = \theta_{fut}\) 0.090 0.025

Update of DDCP when not stopping the trial in two blinded interim analyses

The theory developed in Rufibach et al. (2016) can straightforwardly be extended to accomodate more than one interim analysis. This is illustrated in the tutorial for interim updates for a T2E endpoint.

References

Rufibach, K., P. Jordan, and M. Abt. 2016. “Sequentially updating the likelihood of success of a Phase 3 pivotal time-to-event trial based on interim analyses or external information.” J Biopharm Stat 26 (2): 191–201. http://dx.doi.org/10.1080/10543406.2014.972508.