Skip to contents

Consistency of the predictive distribution model

Source: R/consistency.r
consistency.Rd

Calculate consistency (DCPP, DCP) of the model as the difference of the confidence calculated on the evaluation and the confidence calculated on the training subset. Consistency serves as a proxy for model's transferability.

Usage

consistency(conf_train, conf_eval)

Arguments

conf_train

Confidence calculated on the training subset: a numeric vector of length one, containing a number within the [0, 1] interval. Typically calculated by function confidence() using the training subset.

conf_eval

Confidence calculated on the evaluation subset: a numeric vector of length one, containing a number within the [0, 1] interval. Typically calculated by function confidence() using the evaluation subset.

Value

A numeric vector of length one. It is either NA_real_ or a number within the [-1, 1] interval. Typically, it falls within the

[-1, 0] interval. Greater value indicates more consistent/transferable model. I.e, the closer the returned value is to -1, the less consistence/transferable the model is. Value above 0 might be an artifact or might indicate that the training and evaluation subsets were accidentally swapped.

See also

thresholds for calculating the two thresholds, confidence for calculating confidence

Examples

# Simple examples:
consistency(conf_train = 0.93,
            conf_eval = 0.21) # -0.72 - hardly consistent/transferable model
#> [1] -0.72
consistency(conf_train = 0.43,
            conf_eval = 0.35) # -0.08 - consistent/transferable model, although not so confident
#> [1] -0.08
consistency(conf_train = 0.87,
            conf_eval = 0.71) # -0.16 - a consistent/transferable model that is confident as well
#> [1] -0.16
consistency(conf_train = 0.67,
            conf_eval = 0.78) # 0.11 - positive value might be an artifact
#> [1] 0.11
consistency(conf_train = 0.67,
            conf_eval = NA_real_) # NA
#> Warning: Parameter 'conf_eval' is expected to fall within the [0, 1] interval, but found to be NA.
#> [1] NA

# Real-life case:
set.seed(12345)
observations <- c(rep(x = FALSE, times = 500),
                 rep(x = TRUE, times = 500))
predictions <- c(runif(n = 500, min = 0, max = 0.7),
                 runif(n = 500, min = 0.3, max = 1))
dataset <- data.frame(
  observations = observations,
  predictions = predictions,
  evaluation_mask = c(rep(x = FALSE, times = 250),
                      rep(x = TRUE, times = 250),
                      rep(x = FALSE, times = 250),
                      rep(x = TRUE, times = 250))
)
thresholds_whole <- thresholds(observations = dataset$observations,
                               predictions = dataset$predictions)
confidence_training <- confidence(observations = dataset$observations[!dataset$evaluation_mask],
                                  predictions = dataset$predictions[!dataset$evaluation_mask],
                                  thresholds = thresholds_whole) # 0.602
confidence_evaluation <- confidence(observations = dataset$observations[dataset$evaluation_mask],
                                    predictions = dataset$predictions[dataset$evaluation_mask],
                                    thresholds = thresholds_whole) # 0.520
consistency(conf_train = confidence_training,
            conf_eval = confidence_evaluation) # -0.083 - consistent/transferable model
#> [1] -0.08302792

# Wrong parameterization:
try(consistency(conf_train = 1.3,
                conf_eval = 0.5)) # warning
#> Warning: Parameter 'conf_train' is expected to fall within the [0, 1] interval, but found to be 1.300.
#> [1] -0.8
try(consistency(conf_train = 0.6,
                conf_eval = c(0.4, 0.5))) # warning
#> Warning: Parameter 'conf_eval' has more elements (2) then expected (1). Only the first element is used.
#> [1] -0.2