| Title: | Weight of Evidence for Quantifying Performance of a Binary Classifier |
|---|---|
| Description: | The distributions of the weight of evidence (log Bayes factor) favouring case over noncase status in a test dataset (or test folds generated by cross-validation) can be used to quantify the performance of a diagnostic test. This package can be used with any test dataset on which you have computed prior probabilities of case status, posterior probabilities of case status, and you have the observed case-control status. In comparison with the C-statistic (area under ROC curve), the expected weight of evidence (expected information for discrimination) has several advantages as a summary measure of predictive performance. To quantify how the predictor will behave as a risk stratifier, the quantiles of the distributions of weight of evidence in cases and controls can be calculated and plotted. |
| Authors: | Paul McKeigue [aut, cre], Marco Colombo [ctb] |
| Maintainer: | Paul McKeigue <[email protected]> |
| License: | GPL-3 |
| Version: | 0.7.0 |
| Built: | 2026-05-12 07:01:04 UTC |
| Source: | https://github.com/pmckeigue/wevid |
This package provides functions for quantifying the performance of a diagnostic test (or any other binary classifier) by calculating and plotting the distributions in cases and noncases of the weight of evidence favouring case over noncase status.
To use it, you should have computed on a test dataset (or on test folds used for cross-validation:
1. The prior probability of case status (this may be just the frequency of cases in the training data.
2. The posterior probability of case status (using the model learned on the training data to predict on the test data)
3. The observed case status (coded as 0=noncase, 1=case).
Paul McKeigue [email protected]
Citation for the statistical methods used in this package: McKeigue P. Quantifying performance of a diagnostic test as the expected information for discrimination: relation to the C-statistic. Statistical Methods for Medical Research 2018, in press.
Compute area under the ROC curve according to model-based densities
auroc.model(densities)auroc.model(densities)
densities |
Adjusted densities computed by
|
The area under the model-based ROC curve computed from the densities of the weight of evidence in cases and noncases. This model-based ROC curve is always concave (if the densities have been adjusted to make them mathematically consistent).
plotroc
data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) auroc.model(densities.adj)data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) auroc.model(densities.adj)
Example dataset based on cross-validated prediction of outcome in the Cleveland Heart Study
clevelandcleveland
A data frame with 297 rows and three variables
prior probabilities of case status
posterior probabilites of case status
case-control status
Compute the expected information for discrimination (expected weight of evidence) from the model-based densities
lambda.model(densities)lambda.model(densities)
densities |
Adjusted densities computed by
|
The model-based expected information for discrimination.
data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) lambda.model(densities.adj)data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) lambda.model(densities.adj)
Means of densities in cases and controls
means.densities(densities)means.densities(densities)
densities |
Adjusted densities computed by
|
numeric vector of length 2: mean densities in controls and in cases.
Plot the cumulative frequency distributions in cases and in controls
plotcumfreqs(densities)plotcumfreqs(densities)
densities |
Adjusted densities computed by
|
data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotcumfreqs(densities.adj)data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotcumfreqs(densities.adj)
Plot crude and model-based ROC curves
plotroc(densities, y, W)plotroc(densities, y, W)
densities |
Adjusted densities computed by
|
y |
Binary outcome label (0 for controls, 1 for cases). |
W |
Weight of evidence (natural logs). |
ggplot of crude and model-based ROC curves
data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotroc(densities.adj, cleveland$y, W)data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotroc(densities.adj, cleveland$y, W)
plot log case/control density ratio against weight of evidence as a check that the densities are mathematically consistent
plotW(densities, W)plotW(densities, W)
densities |
Adjusted densities computed by
|
W |
Weight of evidence. (natural logs) |
ggpplot of natural log case/control density ratio against weight of evidence (should be a straight line of gradient 1 passing through the origin)
data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotW(densities.adj, W)data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotW(densities.adj, W)
Plot the distribution of the weight of evidence in cases and in controls
plotWdists( Wdensities.unadj, Wdensities.adj, mask = NULL, distlabels = c("Crude", "Adjusted") )plotWdists( Wdensities.unadj, Wdensities.adj, mask = NULL, distlabels = c("Crude", "Adjusted") )
Wdensities.unadj |
Unadjusted densities computed by
|
Wdensities.adj |
Adjusted densities computed by
|
mask |
if not null, breaks y axis to show more detail of lower end |
distlabels |
Character vector of length 2 |
data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotWdists(densities.unadj, densities.adj)data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotWdists(densities.unadj, densities.adj)
Proportions of cases and controls below a given threshold of W (natural logs)
prop.belowthreshold(densities, w.threshold)prop.belowthreshold(densities, w.threshold)
densities |
Adjusted densities computed by
|
w.threshold |
Threshold value of weight of evidence (natural logs). |
numeric vector of length 2: proportions of controls and cases with weight of evidence below the given threshold.
data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) w.threshold <- log(4) # threshold Bayes factor of 4 prop.belowthreshold(densities.adj, w.threshold)data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) w.threshold <- log(4) # threshold Bayes factor of 4 prop.belowthreshold(densities.adj, w.threshold)
Adjust the crude densities of weights of evidence in cases and controls to make them mathematically consistent
Wdensities.fromraw(densities)Wdensities.fromraw(densities)
densities |
Unadjusted densities computed by
|
data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotWdists(densities.unadj, densities.adj)data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotWdists(densities.unadj, densities.adj)
Compute smoothed densities for a spike-slab mixture distribution
Wdensities.mix(y, W, in.spike, range.xseq = c(-25, 25), x.stepsize = 0.01)Wdensities.mix(y, W, in.spike, range.xseq = c(-25, 25), x.stepsize = 0.01)
y |
Binary outcome label (0 for controls, 1 for cases). |
W |
Weight of evidence. |
in.spike |
logical vector same length as y, TRUE if in spike component, FALSE otherwise. Typically used where high proportion of values of the predictor are zero |
range.xseq |
Range of points where the curves should be sampled. |
x.stepsize |
Distance between each point. |
Calculate the unadjusted smoothed densities of W in cases and in controls
Wdensities.unadjusted( y, W, range.xseq = c(-25, 25), x.stepsize = 0.01, adjust.bw = 1 )Wdensities.unadjusted( y, W, range.xseq = c(-25, 25), x.stepsize = 0.01, adjust.bw = 1 )
y |
Binary outcome label (0 for controls, 1 for cases). |
W |
Weight of evidence. |
range.xseq |
Range of points where the curves should be sampled. |
x.stepsize |
Distance between each point. |
adjust.bw |
Bandwidth adjustment. |
unadjusted density object
Calculate weights of evidence in natural log units
weightsofevidence(posterior.p, prior.p)weightsofevidence(posterior.p, prior.p)
posterior.p |
Vector of posterior probabilities generated by using model to predict on test data |
prior.p |
Prior probabilities on test data. |
The weight of evidence in nats for each observation.
data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotWdists(densities.unadj, densities.adj)data("cleveland") # load example dataset W <- with(cleveland, weightsofevidence(posterior.p, prior.p)) densities.unadj <- Wdensities.unadjusted(cleveland$y, W) densities.adj <- Wdensities.fromraw(densities.unadj) plotWdists(densities.unadj, densities.adj)
Summary evaluation of predictive performance
wtrue.results(studyname, y, posterior.p, prior.p)wtrue.results(studyname, y, posterior.p, prior.p)
studyname |
Name of the study. |
y |
Binary outcome label (0 for controls, 1 for cases). |
posterior.p |
Vector of posterior probabilities generated by a logistic regression model. |
prior.p |
Vector of prior probabilities. |
A dataframe listing some metrics of predictive performance.
data("cleveland") # load example dataset with(cleveland, wtrue.results("cleveland", y, posterior.p, prior.p))data("cleveland") # load example dataset with(cleveland, wtrue.results("cleveland", y, posterior.p, prior.p))