Based on the PIRD strategy (see Table 1), the following question was raised: “What is the calibration and discriminating capacity of the mortality prognostic scales in oncological patients in the ICU?

A selection was made of studies involving patients over 18 years of age with an active cancer diagnosis (regardless of stage) admitted to the ICU. Pregnant patients were excluded. With regard to the types of studies, we included retrospective and prospective cohort studies, with no language restrictions. Articles not containing information on the performance of a prognostic model of mortality were excluded.

A systematic literature search without language restrictions was made of the following electronic databases in October 2022: PubMed, Scopus, Virtual Health Library (VHL) and medRxiv. The following MeSH terms were used, differentiated according to the components of the research question: “Neoplasms”, “Oncology”, “Critical Illness”, “Intensive Care Units”, “Mortality”, “Mortality Prediction”, “Hospital Mortality”, “APACHE”, “Simplified Acute Physiology Score”, “Organ Dysfunction Scores”, “Prognostic Score”, “Outcome prediction” and “Predictive models” (Table 2). In addition, a snowball search strategy was adopted that allowed the inclusion of two additional relevant articles that met the selection criteria. A total of 12 studies were finally included in the systematic review. The present study was registered in PROSPERO (CRD42023441046 – available at https:// https://www.crd.york.ac.uk/prospero/#recordDetails).

The articles of the initial search were independently evaluated by four investigators on a blind basis. The studies were first assessed from the title and abstract, and the selected articles were then subjected to full-text evaluation, with the application of the inclusion and exclusion criteria. There was no specified discrimination according to the index test and reference model in the global studies. Observed mortality was therefore regarded as the reference test for comparison, and the index tests corresponded to the evaluated scales (Fig. 1).

Figure 1.

Article selection process according to the PRISMA guidelines for the reporting of systematic reviews and meta-analyses.

(0.29MB).

Two investigators independently extracted and entered the information from the studies into an Excel spreadsheet. The instrument was constructed by consensus among the investigators, with the variables of interest, ordered chronologically by date of publication. The studies were analyzed by number of patients and type of cancer (solid, hematological and metastatic [advanced disease]). Data reporting was standardized in the form of percentages (Table 3). The discrimination and calibration values of each scale were obtained, and the observed and predicted mortality data were used to calculate the standardized mortality ratio (SMR) (Table 4).

External validity was assessed by analysis of the study population, generalization and random error. Internal validity in turn was assessed by determining information, selection and confounding bias. Evaluation of the methodological quality and risk of bias of the primary studies included in the systematic review was made by paired application of the QUADAS-2 (Quality Assessment of Diagnostic-Accuracy Studies) tool (Fig. 2), specifically designed for assessing the quality of diagnostic test studies. According to our criteria, good study quality was defined by a risk of bias of under 25%, while moderate quality corresponded to 25–50%, and low quality was defined by a risk of bias of over 50%.

Figure 2.

Evaluation of the methodological quality of the included studies using the QUADAS-2 checklist. (a) Risk of bias and applicability plot: Author assessment related to each domain and presented as a percentage in all the included studies. (b) Risk of bias: Author assessment related to each domain for each study.

(0.58MB).

Major differences between the studies were observed, with important variability regarding the number of patients included, direction in time, classification of the type of cancer, and the mortality scales evaluated. Thus, in this stage of the review, the studies were considered to be too heterogeneous to be combined in a meta-analysis, and the decision was made to only carry out a qualitative systematic review of the literature. The narrative synthesis was established from the evaluation of the operative characteristics of the mortality scales and the most significant outcomes.

The results of the methodological evaluation of the studies included in the systematic review are shown in Fig. 2. In general, the studies presented good methodological quality and the observed risk of bias was low. The main problem of bias in the evaluation corresponded to the application of the index test since most of the studies failed to specify cut-off points for the interpretation of discrimination and calibration. The studies with the best methodological quality were those of Schellongowski7 and Kopterides,12 while the poorest quality corresponded to the studies of Pérez,15 Rojas14 and Xue Zhong Xing.13 The remaining articles, published by Sculier,6 Berghmans,8 Soares 2004,9 Park,10 Soares 2009 11, Siddiqui16 and Martos-Benítez,17 were of moderate quality. Good applicability was consistently observed in all the studies.

The studies evaluated the predictive capacity of the scales through discrimination of the area under the curve (AUC) and calibration using the Hosmer-Lemeshow goodness of fit for H or Ĉ, interpreted with the p-value. Discrimination measures the probability that a patient who died had a higher predicted mortality risk than a patient who survived. An AUC = 0.5 indicates that there are no differences between applying the scale or not, while AUC = 1 indicates that the model is perfect, AUC > 0.8 indicates that the model is adequate, AUC = 0.6 to 0.8 indicates that the model is moderate, and AUC < 0.6 indicates that the model is deficient. Calibration in turn compares the probabilities referred to predicted mortality and observed mortality. The selected studies used the Hosmer-Lemeshow goodness of fit for H or Ĉ test, assessed with the p-value, where p < 0.05 indicates a significant difference between observed mortality and predicted mortality, while p > 0.05 suggests an estimate that is closer to reality.

Use was made of the standardized mortality rate (SMR) to compare observed and predicted mortality. In this regard, SMR = 1 indicates that the model is perfect, while SMR < 1 implies that the model overestimates mortality, and SMR > 1 indicates that the model underestimates mortality (Table 4).

APACHE II. This scale was evaluated in 9 of the articles,6–10,12,13,15,17 of which 7 were prospective cohort studies6–10,12,17 and two were retrospective studies.13,15 The number of patients ranged between 126 and 5767, corresponding to the publications of Kopterides12 and Park 2009.10 Patients with solid tumors, advanced disease and hematological malignancies were included. Low to good discrimination was observed, with AUC between 0.60 and 0.96, in the studies of Sculier 20006 and Martos-Benitez 2020.17 Most of the studies showed poor calibration with SMR (>1) between 1.06–1.54, which resulted in uniform underestimation of mortality risk. Two studies reported SMR < 1, corresponding to the largest studies, published by Park 200910 and Xing 2015,13 respectively.

APACHE III. This scale was not taken into account in the conclusions of the review, since it was only evaluated in a single article published by Soares 20049 – a prospective observational cohort study with 542 patients. Solid tumors and hematological malignancies were included. Good discrimination was reported, with AUC = 0.81. Calibration yielded SMR = 1.38, indicative of the underestimation of mortality.

APACHE IV. This scale was evaluated in three of the articles,13–15 of which two were prospective observational cohort studies13,14 and one was a retrospective study.16 The number of patients ranged between 283–981, corresponding to the publications of Rojas 201814 and Xing 2015.13 Patients with solid tumors, advanced disease and hematological malignancies were included. Moderate to good discrimination was observed, with AUC between 0.73 and 0.95. Calibration yielded SMR = 1.17–1.36, indicative of the underestimation of mortality.

No significant differences were observed indicating superior prognostic performance among the different versions of the APACHE scale (II, III and IV). The three models may underestimate mortality in oncological patients.

SOFA. This scale was evaluated in two of the articles,12,15 of which one involved a prospective design12 while the other was a retrospective study.15 An average of 129 patients were included of which over 94% had solid tumors. The study carried out by Kopterides 201112 showed good discrimination capacity, with AUC = 0.83, while the study published by Pérez 201915 presented moderate discrimination, with AUC = 0.77. Both studies showed low calibration, with p < 0.22, and both underestimated mortality, with SMR > 1.23.

SAPS II. This scale was evaluated on a prospective basis in 7 studies,6–12 four of which involved less than 500 patients,6–8,12 while the remaining three involved over 500 patients.9–11 Solid tumors and hematological malignancies were included. Four studies showed good discrimination, with AUC > 0.8,7,9,11,12 while three studies presented moderate discrimination, with AUC < 0.72.6,8,10 Four articles reported low calibration, with p < 0.05,6,8,10,11 while two presented p > 0.05,7,12 and one article failed to supply this information.9 Almost all of the publications reported SMR > 1.1, except the study by Schellongowski in 2003,7 which did not supply this information, resulting in the underestimation of mortality.

SAPS III. This scale was evaluated in four studies,11,13,16,17 of which three presented a prospective design11,16,17 and one was a retrospective study,13 with a sample size of between 414 and 981 patients, and a predominance of solid tumors (>50%). Three studies11,13,17 reported good discrimination, with AUC = 0.84, 0.91 and 0.95, respectively, while one study presented moderate discrimination capacity, with AUC = 0.7.16 Two articles presented low calibration,11,17 with p < 0.05, while the other two showed good calibration, with p > 0.05. All four studies evidenced an underestimation of mortality, with SMR = 1.08 to 1.50.

SAPS III – CSA. The study carried out by Martos-Benítez 2020,17 a prospective cohort study involving 414 patients, mostly with solid tumors (94.9%), evaluated the SAPS III in the context of Central and South America (SAPS III - CSA). The study showed good discrimination, with AUC = 0.95, and good calibration, with p > 0.05. Mortality was overestimated, with SMR = 0.85.

ICMM. This scale was evaluated in 6 studies,7–9,11,15,16 of which 5 presented a prospective design and one was a retrospective study,15 with a sample size of between 132 and 1257 patients, corresponding to the articles published by Pérez 201815 and Soares 2010,11 respectively. Patients with solid tumors, advanced disease and hematological malignancies were included, except the study published by Pérez 2018,15 which only included solid tumors. Four articles9,11,15,16 reported SMR (<1) = 0.47 to 0.83, indicative of the overestimation of mortality, while two underestimated mortality.7,8 The study carried out by Schellongowski 20037 showed the best calibration, with SMR = 1.05.

MPM II. The MPM II scale applied upon admission and after 24 h was evaluated in three studies,9,14,16 all of which involved a prospective design. The number of patients ranged between 283 and 1257, corresponding to the studies of Rojas 2018 and Soares 2004, respectively. Patients with solid tumors, advanced disease and hematological malignancies were included. Moderate discrimination was observed, with AUC = 0.67 and 0.79, in the studies of Siddiqui 202016 and Soares 2004,9 respectively. In all three studies, SMR was >1, indicative of underestimation of mortality.

MPM III. This scale was only evaluated in the article published by Soares 2010,11 a prospective cohort study involving 717 patients with solid tumors and hematological malignancies. Moderate discrimination was reported, with AUC = 0.71. Mortality was overestimated, with SMR = 3.41.