When users choose to estimate and evaluate ITR under cross-validation, the package implements Algorithm 1 from Imai and Li (2023) to estimate and evaluate ITR. For more information about Algorithm 1, please refer to the this page.
Instead of specifying the split_ratio argument, we
choose the number of folds (n_folds). We present an example
of estimating ITR with 3 folds cross-validation. In practice, we
recommend using 10 folds to get a more stable model performance.
| Input | R package input | Descriptions |
|---|---|---|
| Data \(\mathbf{Z}=\left\{\mathbf{X}_i, T_i, Y_i\right\}_{i=1}^n\) | treatment = treatment, form = user_formula, data
= star_data |
treatment is a character string specifying
the treatment variable in the data; form is a
formula specifying the outcome and covariates; and a dataframe
data |
| Machine learning algorithm \(F\) | algorithms = c("causal_forest") |
a character vector specifying the ML algorithms to be used |
| Evaluation metric \(\tau_f\) | PAPE, PAPD, AUPEC, GATE | By default |
| Number of folds \(K\) | n_folds = 3 |
n_folds is a numeric value indicating the
number of folds used for cross-validation |
| … | budget = 0.2 |
budget is a numeric value specifying the
maximum percentage of population that can be treated under the budget
constraint |
library(evalITR)
# estimate ITR
set.seed(2021)
fit_cv <- estimate_itr(
treatment = treatment,
form = user_formula,
data = star_data,
algorithms = c("causal_forest"),
budget = 0.2,
n_folds = 3)
#> Evaluate ITR with cross-validation ...The output will be an object that includes estimated evaluation metric \(\hat{\tau}_F\) and the estimated variance of \(\hat{\tau}_F\) for different metrics (PAPE, PAPD, AUPEC).
# evaluate ITR
est_cv <- evaluate_itr(fit_cv)
#> Cannot compute PAPDp
# summarize estimates
summary(est_cv)
#> -- PAPE ------------------------------------------------------------------------
#> estimate std.deviation algorithm statistic p.value
#> 1 0.2 0.76 causal_forest 0.26 0.79
#>
#> -- PAPEp -----------------------------------------------------------------------
#> estimate std.deviation algorithm statistic p.value
#> 1 2.5 0.65 causal_forest 3.8 0.00012
#>
#> -- PAPDp -----------------------------------------------------------------------
#> Cannot compute PAPDp
#>
#> -- AUPEC -----------------------------------------------------------------------
#> estimate std.deviation algorithm statistic p.value
#> 1 1.3 1.6 causal_forest 0.83 0.41
#>
#> -- GATE ------------------------------------------------------------------------
#> estimate std.deviation algorithm group statistic p.value upper lower
#> 1 -92.1 119 causal_forest 1 -0.776 0.44 140 -325
#> 2 68.7 72 causal_forest 2 0.955 0.34 210 -72
#> 3 20.4 140 causal_forest 3 0.146 0.88 294 -254
#> 4 14.8 87 causal_forest 4 0.169 0.87 186 -156
#> 5 6.4 66 causal_forest 5 0.097 0.92 136 -123