Improvement in the quality of critical care requires quantitative assessment of the structure used, the care process, and patient evolution.1 In this regard, patient evolution is clearly determined by the initial severity of the disease condition. Measuring severity is a constant concern in the critically ill patient, and offers a range of benefits.2 In effect, the measurement of severity helps establish a prognosis, fundamentally in terms of patient mortality. Furthermore, it allows us to know the profile of our patients, and has an impact upon the management of the material and human resources used. Prognostic assessment, by allowing the adjustment of actual mortality to predicted mortality, affords a care quality indicator that facilitates comparison with other institutions (benchmarking).1 Lastly, the measurement of severity plays an important role in facilitating the investigation of outcomes, by ensuring the comparison of patients that exhibit similar severity factors. In this respect, severity scales have been designed with a view to standardizing the measurement of severity.

Three large groups of severity scales with different versions are predominantly used3: the Acute Physiology and Chronic Health Evaluation (APACHE), the Simplified Acute Physiology Score (SAPS), and the Mortality Prediction Models (MPM). Based on a logit function, these scales transform their scores into in-hospital mortality probability values, grouping the patients in terms of the probability of death. Such predictive scoring systems have been developed from the analysis of large cohorts of patients with different medical–surgical diseases.4 Their main limitations are a lack of individual predictive capacity, their use in patient populations not included in the studies from which the scales were originally developed, or their application to groups of patients with concrete diseases.5

Although many factors are related to mortality (age, comorbidity, diagnosis upon admission to the Intensive Care Unit [ICU], etc.), the severity of the physiological anomalies are its fundamental conditioning parameter.6 In the critical patient, these anomalies find their maximum expression in multiple organ dysfunction syndrome (MODS). Patients with MODS present a high mortality rate that is correlated to the number of affected organs.7 Although these individuals form part of the population of patients from which the severity scales were developed (accounting for about one-third of the total patients in a third-level ICU), we have found no previous studies exclusively based on subjects of this kind.4

The present study was carried out to determine the degree to which the currently most widely used predictive scoring systems are able to predict real mortality in the ICU among patients who develop MODS in the first 24h of admission.

A retrospective cohort study was made in the ICU of Hospital Virgen de la Salud (Toledo, Spain), consecutively analyzing the case histories of medical–surgical patients with MODS over a period of two years. The patients were required to be over 16 years of age, and with a minimum ICU stay of 12h. We defined MODS as the presence of at least two dysfunctional organs during the first 24h of admission to the ICU, as assessed by the SOFA scale.8 All patients with missing data needed for calculation of the severity scores were excluded, as were patients referred from other ICUs with an evolutive course of over 24h, cases of brain death, and patients readmitted to the ICU during the same hospital stay.

The patients were followed-up on during their stay in the ICU and in the hospital ward until discharge or death – with registry of the final outcome. General data corresponding to admission were collected (gender, age, presence of infection, basal situation – asymptomatic for activities of daily living – origin, diagnosis upon admission, need for mechanical ventilation, destination at discharge from the ICU, hospital and ICU stays, death and place of death), together with the information needed for the APACHE II3 and IV,9 SAPS II10 and III11 (European scale) and MPM II severity scales.12 The MPM II employed was that assessed after 24h, in order to allow comparison with the SAPS and APACHE scores, since all three are assessed within the first 24h.

Quantitative data were reported as the mean (standard deviation [SD]), and categorical data as count (percentage). The comparison of categorical variables was based on the chi-squared test, with the Fisher approximation for 2×2 tables where indicated. The Student t-test in turn was used for the comparison of quantitative variables.

We calculated all the scores of the different scales and the predicted mortality figures. Discrimination and calibration were studied to evaluate the validity of the model.

Discrimination is the capacity to distinguish between subjects that die and those that survive, describing the precision of a given prediction. For example, if the model predicts a mortality rate of 35%, its predictive capacity is perfect if the observed mortality rate is 35%. The assessment of discrimination was made based on the receiver operating characteristic (ROC) curve (95% confidence interval [95%CI]), and the area under the curve (AUC) can be regarded as a global measure of the discriminating capacity of the model.

Calibration of the model is the adjustment or fit that assesses concordance (agreement) between the probabilities observed in the sample and the probabilities predicted by the model – describing how the prognostic scale is executed over a broad range of predicted mortalities. Thus, good calibration is precise in mortalities of 10%, 50% and 80%, for example. Calibration was assessed based on the Hosmer–Lemeshow goodness-of-fit test, which exhibits the best fit when the probability is equal to 1 (non-significance indicating better fit).13 In this respect, the mortality probabilities were distributed into 10 defined mortality probability groups (risk deciles), attempting to categorize an approximate number of patients in each group.

The assessment of calibration was complemented by the standardized mortality ratio (SMR), i.e., the ratio between observed mortality and predicted mortality. If SMR=1, calibration is regarded as good, with perfect agreement, since observed mortality equals predicted mortality. In contrast, if SMR<1, the mortality is overestimated, while SMR>1 means that mortality is underestimated.

The data were analyzed using the SPSS® version 20 statistical package (IBM Corp., 2011). Statistical significance was considered for p<0.05.

The study population during the evaluated period comprised 568 patients that met the inclusion criteria and none of the exclusion criteria. Table 1 reports the most frequent diagnoses. The global in-hospital mortality rate was 39.8% (226 patients); 165 (29.1%) died in the ICU, and 61 (10.7%) in the hospital ward following discharge from the ICU. The characteristics referred to admission, the patient course and severity scales are shown in Table 2.

Calibration, evaluated for the severity scales using the Hosmer–Lemeshow goodness-of-fit test, is shown in Table 3. Fig. 1 evaluates the different ROC curves according to the established predictive model, and Table 4 shows the corresponding areas under the curve, with the 95%CI and significance for each scale. Table 5 in turn determines the SMR for each predictive scoring system with the corresponding 95%CI. Fig. 2 shows the different calibration curves.

Figure 1.

Assessment of discrimination based on the receiver operating characteristic (ROC) curve for the different predictive models.

(0.12MB).

Figure 2.

Calibration curves with the 95% confidence intervals (95%CI) for observed mortality (line with square markers, with dashed lines). The lines without markers represent predicted mortality; the lines with round markers represent the percentage of subjects for each of the mortality deciles.

(0.55MB).

In our series we included patients with MODS in the first 24h of admission. This is the period usually considered by the predictive scoring systems for assessing the physiological data, since it coincides with the interval of greatest physiological deterioration. Although the predictions that use physiological data in the first hour of admission have revealed no statistically significant differences with the worst values of the first 24h,14 fewer missing values and a greater explanatory capacity are obtained when considering a period of 24h.15

Our series showed a high in-hospital mortality rate, with the death of almost four out of every 10 patients, particularly in the period of admission to the ICU, with long ICU stays (over one week), post-ICU hospital stays (about 2 weeks), a high frequency of mechanical ventilation during admission (about 79%), and high dysfunction predictive scores (SOFA score of about 8). This coincides with the findings of other series involving patients in these evolutive stages and levels of severity.16

The most frequent diagnoses were consistent with those found in any polyvalent (medical–surgical) ICU. The patients that died presented an already known profile, being of older age, admitted from the hospital ward, with a greater incidence of mechanical ventilation, a longer ICU stay, a shorter post-ICU stay (possibly because of death), a larger number of dysfunctional organs (almost 30% more), and with poorer predictive scores.17 A more detailed evaluation of the validity of the models requires an analysis of discrimination and calibration. In this regard, it must be remembered that poor calibration and discrimination may be due to differences in the quality of expected care, but also to poor applicability of the model to a concrete patient population.18

The APACHE, SAPS and MPM are the three most widely used scales. Problems referred to ease of use, the number of required variables (142 in the case of APACHE IV, 20 in SAPS III, and 13 in MPM II 24h, for example), and the costs associated to their use have limited the application of the most recent (and improved) versions. Among the different existing version, the APACHE II is the most widely used instrument.19 In this respect, APACHE II exhibited shortcomings in predicting specific patient subgroups (liver failure, sepsis, etc.), and showed predicted mortality to be lower than the observed mortality in ICU patients transferred from other hospital stays (a phenomenon known as lead time bias)20,21 (this being the reason why we excluded patients transferred from other centers). A similar situation may occur with the SAPS II, which is more widely used than the SAPS III. This circumstance would reflect not only the limitations inherent to its lesser complexity (the use of “less refined” versions), but also the fact that we would be comparing our patients with cohorts already over two decades old. All the severity scales require periodic revisions, recalibrations and updates in relation to the improvement of patient care (treatment and other factors that influence mortality, such as the introduction of new technologies). Furthermore, these revisions should be used in the contemporaneous studies carried out. Failure to update the predictive scoring systems would result in gradual loss of calibration over time, with the overestimation of mortality for any given severity scale.22,23 This over-prognosis effect was not clearly seen in our study.

We have used the most important principles required for assessing a severity scale, employing adequate discrimination and calibration to effectively measure the prognosis referred to in-hospital mortality. The predictive scoring systems generally showed excellent discrimination–the main problem being shortcomings in calibration. This was particularly manifest in the case of the APACHE scale, but was also seen in the rest of the predictive scoring systems (SAPS III, etc.).22

There are important differences among the range of predictive scoring systems in relation to data collection, the calculation of mortality, efficacy and costs.24 While the APACHE scale requires the documentation of many physiological and general data, the other instruments use fewer variables and have a simpler coding system, which facilitates data collection.25 The transition from simplified predictive systems to other more complex instruments is a long process, with many problems. In this respect, developments in computer science have made it possible to use mathematically complex predictive systems, though this has not been accompanied by parallel developments in the automation of data recording.14 All this complicates data collection in predictive scoring systems such as the APACHE IV, where complexity is maximum. In contrast, other systems such as the MPM or SAPS, place emphasis on simplicity. Few studies have compared the three predictive scoring systems; the APACHE IV generally exhibits better predictive precision, though the MPM (II and III) has been shown to be an effective alternative when priority is focused on lower costs and a lesser complexity of the collected data.25,26

There are no previous studies on the use of predictive scoring systems in patients only with MODS, though such patients to one degree or other form part of all the series published in the intensive care setting. Indeed, the opposite scenario is often observed: for example, about two-thirds of the APACHE IV validation cohort presented low risk at the time of admission (mortality rate<10%).27,28

The best performance in terms of discrimination corresponded to the APACHE IV, which yielded adequate values (0.805). In comparison, SAPS II (0.755), MPM II (0.748) and SAPS III (0.737) showed progressively poorer discrimination. The APACHE II scale showed the poorest discrimination of all the instruments (0.699).

In our series, MPM II was the predictive model with the best calibration, followed by SAPS III. The APACHE II, SAPS II and APACHE IV scales had very poor calibration. Specifically, the greatest shortcomings in terms of calibration corresponded to the APACHE IV scale.

In relation to SMR, all the scales showed values of close to 1, except the APACHE IV scale, which yielded 1.9 (95%CI 1.78–2.02) and thus more clearly underestimated mortality. In general terms, APACHE II and SAPS III tended to underestimate mortality, while MPM II tended to overestimate mortality, though in all cases the deviations were discrete. The SMR is a mandatory clinical management parameter in some countries in both Europe and the United States (all the ICUs belonging to the veterans administration), and it is usually used as an international quality measure29–though some authors express concern about exclusive use of the SMR as a quality indicator.14 Many users simply seek to achieve SMR<130. However, despite its widespread use, the SMR is well known to have some intrinsic limitations.30 In this regard, high-risk patients contribute disproportionately to SMR, since many of these individuals die. It is therefore important to report SMR separately for patients with low, moderate and high risk. In this way the impact of a disproportionate number of high-risk patients becomes diluted, and the SMR is no longer so dependent upon the type of patients seen in each ICU. This is important, since a growing number of studies speak of poorer calibration precision at the extremes of severity.9,27,31

On examining the calibration curves we can establish a moderate mortality risk group (in-hospital mortality rate 10–50%) and a high mortality risk group (in-hospital mortality rate >50%), which can be analyzed separately.27 The APACHE IV was the predictive scale that most intensely underestimated predicted mortality in the moderate risk group. The same situation was observed in the high risk group, though to a lesser degree. The same pattern, albeit of lesser magnitude, was recorded with the APACHE II and SAPS II, in both mortality groups. The SAPS III only tended to underpredict in the moderate risk group, and performed well in the high risk group. The MPM II scale showed a different behavior, characterized by over-prediction, particularly in the high risk group. Curiously, in most of the published patient cohorts, in relation to the highest predicted mortality figures, the observed mortality was found to be lower than calculated mortality.18 The calibration curves show the 95% confidence intervals to be very wide (lesser precision); more patients are therefore probably needed in order to obtain more adjusted and precise results.

Our study has limitations. The sample size, while moderate, is not optimum for studies of this kind. Furthermore, this study was carried out in a single center in Spain, outside the geographical setting in which some of the examined scales were validated – a fact that has an impact upon the precision of the model.32–34 On the other hand, the utilization of severity scales that group patients into homogeneous disease processes–while of growing interest–is also a clear limitation.

It is unavoidable (and for many authors desirable) for the predictive scoring systems to become increasingly used, to the point of governing our clinical activities.35 It is important to seek maximum closeness between the real world and the proposed model. In this sense it is crucial to evaluate the tools we use in our patients, and doing so in the individuals of greatest disease severity is of great interest, since we tend to underestimate actual mortality. Each country should ensure that the model fits its own data, since good calibration of the model will afford reliable meaning and adequate estimated probabilities.36 Considering the lack of a locally calibrated system, we have established indirect comparisons of the observed outcome versus the outcome predicted by the different models, applied in the patient populations used to develop the risk model. It is important to do this in the population of interest, in order to implement quality assessment using evolution in the ICU adjusted for risk–evaluating not only mortality but also stays, readmissions to the ICU, or other evolutive parameters.37,38

In our series of patients with MODS, the APACHE IV scale afforded the best discriminative capacity, and we confirmed its poor calibration. The MPM III was not tested, though we found the MPM II to offer good discrimination and the best calibration of all the evaluated scales. The SAPS II in turn offered the second best discrimination, with poor calibration. The APACHE II calibration and discrimination values currently disadvise its use. Lastly, the SAPS III showed good calibration with modest discrimination. We believe that these results could serve to initiate new larger and multicenter studies in certain critically ill populations.

The present study has received no partial or full financial support.

The authors declare that there are no financial or personal interests that may bias or influence their actions in relation to this study.