Evaluation¶

All algorithms submitted to the PRISM-AI Challenge are evaluated using standardized metrics for overall performance, discrimination, calibration and clinical utility, following current recommendations for the performance evaluation of predictive AI models [1].

Everything on this page was fixed in advance. All evaluation scripts are available on GitHub, so that participants can inspect exactly how their algorithm is scored — and, if they wish, reproduce the computation themselves.

Primary analysis¶

The primary outcome is 5-year breast cancer risk, including both invasive breast cancer and DCIS, in the 72,076 participants described on the Data page. The results published on this platform are those of the primary analysis: overall performance, discrimination, calibration and clinical utility for algorithms that output a 5-year breast cancer risk estimate.

Overall performance¶

Density plots show the distributions of predicted risks for participants with and without breast cancer within 5 years. These make visible how much separation an algorithm achieves, and over what range of predicted risk it operates.

Discrimination¶

Discrimination is assessed using Uno's time-dependent area under the receiver operating characteristic curve (tAUC) for right-censored survival data [2]. Because event times may be right-censored, Uno's estimator applies inverse probability of censoring weighting (IPCW) to account for this. The censoring distribution is estimated using the Kaplan–Meier estimator, and the resulting weights are used to obtain an unbiased estimate of the tAUC under the assumption of independent censoring.

Uno's tAUC is calculated over all unique pairs i and j as:

where r_i is the predicted risk score for individual i, T_i is the observed time for individual i, I(·) is the indicator function (1 if true, 0 otherwise), and w_ij is the pair-specific IPC weight.

In addition, an ROC curve is constructed by evaluating sensitivity and specificity of the predicted risk score across possible thresholds.

Calibration¶

Calibration-in-the-large is assessed using the observed-to-expected (O/E) ratio, which compares the observed number of events with the total number of events expected according to the algorithm at 5 years [3]. The observed number of events is estimated using the Kaplan–Meier estimator to account for right censoring; the expected number is obtained by summing the predicted risks across all individuals.

Calibration slope. Calibration is further assessed by a time-point-specific calibration slope, estimated with inverse probability of censoring weights. Risk predictions are transformed and entered into an IPC-weighted regression model, whose coefficient represents the calibration slope [4,5]. A slope of 1 indicates predicted risks that are neither too extreme nor too flat; a slope below 1 indicates that risk estimates are more extreme than the observed risks.

Calibration plot. Individual pseudo-values for the Kaplan–Meier event probability are computed for each subject using the jackknife resampling approach [6]. These are plotted against the predicted risk using loess smoothing [4], giving a smooth calibration curve across the range of predicted risk.

Clinical utility¶

Clinical utility is assessed by calculating net benefit across a range of relevant decision thresholds [7]. For 5-year breast cancer risk, decision thresholds between 0.0% and 5.0% are considered. Net benefit (NB) at decision threshold p_t is:

where TP is the number of true positives, FP the number of false positives and n the sample size. To account for censoring, Kaplan–Meier IPC weights are used to compute true and false positive proportions across thresholds. Results are presented as a decision curve.

Uncertainty¶

All metrics are presented with 95% confidence intervals and all plots with 95% confidence bands, obtained from 1,000 bootstrap-resampled estimates.

Leaderboards¶

All metrics and plots described above are published on the PRISM-AI Challenge pages. By default, the leaderboard will rank algorithms according to their discrimination performance, although all metrics will be displayed and it is possible to rank algorithms according their calibration performance as well.

Secondary analyses¶

A number of pre-specified secondary analyses are carried out by the PRISM-AI research team. These will be reported in the publication of this Challenge and not published on this platform:

• Performance of algorithms that output risk over shorter horizons (2, 3 or 4 years), evaluated with the same metrics at the matching horizon.
• Discrimination of a detection algorithm for breast cancer risk, whose output is not an estimate of future risk.
• Performance in subgroups defined by age, breast density and socio-economic position.
• The added value of volumetric breast density alongside the algorithm's predictions.
• Discrimination for clinically relevant breast cancer subtypes, for invasive cancers only, for advanced cancers.
• A sensitivity analysis accounting for misclassification at the index screening examination, which relaxes the exclusion of recalled examinations.
• A sensitivity analysis in which index screening examinations are excluded where in retrospect detectable cancer can be identified, even though they were not recalled at the time.

References¶

1. Van Calster B, Collins GS, Vickers AJ, Wynants L, Kerr KF, Barreñada L, et al. Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance. The Lancet Digital Health. 2025.
2. Uno H, Cai T, Pencina MJ, D'Agostino RB, Wei L-J. On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in Medicine. 2011;30(10):1105–17.
3. Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology. 2016;74:167–76.
4. Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Statistics in Medicine. 2014;33(3):517–35.
5. Pate A, Sperrin M, Riley RD, Peek N, Van Staa T, Sergeant JC, et al. Calibration plots for multistate risk prediction models. Statistics in Medicine. 2024;43(14):2830–52.
6. Andersen PK, Pohar Perme M. Pseudo-observations in survival analysis. Statistical Methods in Medical Research. 2010;19(1):71–99.
7. Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6.