The vast amount of data obtained from a single patient in an Intensive Care Unit (ICU) makes it humanly impossible to organize and interpret it in the required time, especially to predict the outcome of a patient; for this reason, different types of indicators that synthesize multiple physiological and demographic data into a single number that represents the severity of the illness of a patient have been developed from statistical analysis of the data collected for a large number of patients. Some commonly used indicators are severity-of-disease classification systems as APACHE (Acute Physiology and Chronic Health Evaluation), SAPS (Simplified Acute Physiology Score), MPM (Mortality Prediction Model), OASIS (Oxford Acute Severity of Illness Score), SOFA (Sequential Organ Failure Assessment), among others1; in general, there is a score that increases with the risk of patient mortality. These classification systems are used to determine the risk in population studies conducted in an ICU, and provide a method for benchmarking between intensive care units of different hospitals. However, these indicators lack the precision required for use at individual level, they present significant errors at patient data away from the average. For this reason, efforts have been made to increase the performance of these indicators through the use of computational techniques such as machine learning; in recent years, specific models according to groups of patients that shares a common characteristic (like diagnostics, service type or a similarity metric) have been created.2,3

Sepsis is a life-threatening organ dysfunction due to a dysregulated host response to infection. It is an important public health problem, which generates high costs for the health system and carries a high morbidity and mortality (in-hospital mortality ranged from 14.7% to 29.9% in the United States).4

A model that takes into account the peculiarities of sepsis and identify sensitively and early poor patient's outcome, could become a very useful tool to help the clinical group to understand the severity of the disease and could help to the generation of alerts that favor early onset of therapeutic measures, thus helping to improve the prognosis of patients with sepsis admitted to an ICU.

The performance of mortality prediction systems in patients with suspected sepsis, severe sepsis and septic shock have been evaluated in the ICU,5 customized versions for severe sepsis and septic shock of in-hospital mortality classification systems have also been developed,5,6 and even particular models have been created for in hospital mortality prediction of ICU patients with sepsis, severe sepsis and septic shock.7–10

Cited works report better performance than traditionally severity of disease scores and tend to focus on the prediction of in-hospital mortality, however, long-term outcomes from sepsis are poorly understood. Winters et al. concluded that patients with sepsis have ongoing mortality beyond short-term, so the use of 28-day mortality or in-hospital mortality as end points for clinical studies may lead to inaccurate inferences.11 Shankar-Hari and Rubenfeld assert that in the first year following a sepsis episode, approximately 60% of sepsis survivors have at least one rehospitalization episode, which is most often due to infection and one in six sepsis survivors die.12 In 2007 Yende et al. insure that long-term mortality following severe sepsis is high, and fewer than half of patients who experience severe sepsis are alive at 1 year.13 In a different study published in 2016 Yende et al. studied the long-Term Quality of Life Among Survivors of Severe Sepsis and concluded that, among individuals enrolled in the clinical trial who lived independently prior to severe sepsis, one third had died and of those who survived, a further one third had not returned to independent living by 6 months.14

According to all of the above, the main objective of this study is to develop a model that goes beyond the prediction of in-hospital mortality, for this reason, this paper presents the development of a model for the 1-year mortality prediction of sepsis diagnosed patients in an ICU that outperforms commonly used severity-of-disease classification systems. This model would help identify those patients at greatest risk, and will be the first step to detect signs of alarm from a worse outcome beyond the hospital discharge.

For the model based on all the predictors the parameters that presented a better AUROC and an adequate calibration were M (number of trees)=1150, L (number of splits on each tree)=9 and ν (learning rate)=0.01. The predictive power of the SGB model with all the variables over the 1695 admissions of the validation subset were evaluated and an AUROC of 0.8039 (95% confidence interval (CI): [0.8033 0.8045]) was obtained; with this methodology the relative influence of each variable is scaled so that the sum adds 100, with higher numbers indicating stronger influence on the response, the 37 most important predictors for the 1-year mortality are presented in Fig. 4.

Figure 4.

SGB relative importance of the predictors for the 1-year mortality prediction model. Abbreviations: Bun: blood urea nitrogen; Max: maximum; WBC: white blood cell; Min: minimum; SpO2: peripheral capillary oxygen saturation; PaO2/FiO2: partial pressure arterial oxygen and fraction of inspired oxygen ratio; FiO2: fraction of inspired oxygen; Mechanical vent: mechanical ventilation; DABP: diastolic arterial blood pressure; SABP: systolic arterial blood pressure; MABP: mean arterial blood pressure.

(0.18MB).

After applying LASSO regularization process some of the coefficients of the regression are set exactly to zero. Table 3 presents the LASSO selected predictors (with shrinkage parameter λ=0.01288503). For the SGB model with the intersection variables (18 predictors) the AUROC was 0.792 (95% confidence interval (CI): [0.791 0.793]). The other three SGB models were not different in their performance to the model with all the variables.

To benchmark the proposed SGB models, three severity-of-disease classification systems were used to evaluate the 1-year mortality on the same validation subset. The AUROC values for the reference scores were: OASIS 0.631 (95% confidence interval (CI): [0.630–0.632]), SOFA 0.588 (95% confidence interval (CI): [0.587–0.589]) and SAPS2 0.702 (95% confidence interval (CI): [0.701–0.703]). Fig. 5 presents the box plots of the AUROC and the accuracy of the three reference severity of disease scores and the SGB models with all the predictors and the intersection predictors.

Figure 5.

Box plots of AUROC and accuracy for the 1-year mortality on the validation subset. For the proposed SGB models with all variables (140 predictors), and the variables from the intersection between the SGB variable importance selected variables and the LASSO selected variables (18 predictors).

(0.12MB).

The calibration of the proposed SGB models was adequate with p-values of 0.0916 and 0.127 for the model with all the variables and the model with the intersection variables respectively. Goodness of fit was also adequate (p-values for all variables model: 0.1857 and p-values for intersection model: 0.9219).

For SGB model with the intersection predictors, observed versus predicted of numbers of deaths were compared graphically within deciles of increasing probability of the 1-year mortality (Fig. 6), and it is observed that estimated and observed mortality pairs are similar and shows that the number of outcome events is indeed increasing along the probability deciles. The relative importance of the 18 predictors of the intersection models are listed in Table 4. These predictors allow to identify features that could become prognostic markers for the 1-year mortality of the sepsis diagnosed patients within the ICU and could be an input for a new severity score for patients with sepsis in the ICU.

Figure 6.

Comparison of observed versus predicted 1-year mortality in the deciles of predicted mortality based on the SGB model with the intersection variables.

(0.09MB).

Accuracy and AUROC analysis over the validation data indicate that custom mortality prediction models for a specific disease presents a better performance that traditional scores, which could lead to better management of illness within the ICU. Accuracy and AUROC analysis also ratify the complex interdependence among different physiological systems in response to sepsis, because the SGB models are composed of between 450 and 1150 trees, which are difficult to interpret; therefore, it is necessary to develop easy-to-use computer tools that allow these types of models to be implemented within the ICU. This study indicates that it is possible to generate a specific model for the prediction of mortality of patients admitted to an ICU with a diagnosis of sepsis, that includes variables that are now commonly evaluated and been widely used.

SGB variable importance and LASSO methodologies allowed to develop models that preserve the same performance as the one generated with all the predictors but with less predictors. Also the intersection of the predictors selected by the two methods leads to the development of a much simpler model with only 18 predictors and less trees, which also presents good performance.

As expected, older patients are at greater risk in consequence the most important parameter for the outcome is the age. Urine output is used as a marker of acute kidney injury, a disease that is associated with substantial in-hospital mortality, beside this, it is important to note that it is a relatively simple and widely used variable in the ICU that has a high relevance in predicting 1-year mortality with the SGB methodology.

Minimum lactate over the first 24h of the ICU admission is the ninth most important variable for the outcome prediction in this study; Lactate is currently used within the ICU as a diagnostic tool and as a prognostic marker, since the higher the value, the greater the risk of mortality. However, if the lactate of a patient does not reach below a threshold, it will also have a higher mortality risk. For this reason, the minimum lactate during the first 24h must also be analyzed in ICUs. Mean lactate is also considered an important predictor, which agrees with what is reported in the literature, since hyperlactatemia is related with a poor outcome in ICU.34 An elevated blood urea nitrogen (BUN) is associated with increased mortality in critically ill patients.35

The main objective of this work is to present a model for the 1-year mortality prediction of the patients that are admitted in a ICU with a sepsis diagnosis; and shows that the use of ensemble based algorithms (SGB in this study) and the inclusion of statistical descriptors that are not usually taken into account in the traditional severity-of-disease classification systems (for example mean, minimum and maximum values of the same variable), improves the performance of the prediction of prognosis models in patients admitted to an ICU with diagnosis of sepsis, however, this means that the model can only be used after the first 24h of observation.

Other limitations of this study include the fact that it is based on the data taken at only one institution, however, despite the limitation of being single-centered, the main advantages of MIMIC-III are that, right now, it is the only freely accessible critical care database of its kind, the dataset spans more than a decade and it has detailed information about individual patient care that includes time-stamped nurse-verified physiological measurements. For this reasons MIMIC-III (and specially it previous version MIMIC-II) are widely used internationally. For this study in particular, an important advantage is that besides in-hospital mortality, MIMIC-III provides Out-of-hospital mortality dates through the Social Security Administration Death Master File. On the other hand, there have been few validated methods of medical record data extraction for estimating sepsis, particularly in this work, the Angus criteria was used, which is one of the first protocols using administrative data, and was validated by comparing a nurse-driven identification of a population of patients with the clinical syndrome of sepsis, however, Angus criteria has shown to be capable of capture most of the patients with severe sepsis but not exclusively and cohorts identified by different methodologies (for instances Angus criteria and Martin Criteria) yielded widely different patient groups.36

It is also important to note that the criteria used to select patients is based on the ICD codes, which in MIMIC-III are generated for billing purposes at the end of the hospital stay, hence it does not guarantee that patients suffer from sepsis at the time of admission, even so, we evaluate the data of the first 24 hours after admission since we considered that the majority of patients are usually already infected at ICU admission.

The emergence of machine learning techniques in the field of health is a fact. Specifically, in the field of Intensive Care, it is undeniable that the potential for its application is immense. Specifically, the use of assembly algorithms, and in particular the SGB, allows the development of prediction models, which despite being complex show significantly better discrimination than traditional severity of disease scores (Like OASIS, SAPS II or SOFA). This could be explained by the fact that the base models in the SGB algorithm are not fitted independently, but sequentially, this mean that the subsequent predictors are based on the results of previous predictors, moreover, SGB is based on a steepest gradient algorithm which places emphasis on misclassified training data that are close to their correct classification, which reduces the number of misclassified observations and facilitates mortality prediction models to be used at the individual level.

Javier E. García-Gallo contributed in the concepts, design, definition of intellectual content, literature search, experimental studies, data acquisition, data analysis, statistical analysis, manuscript preparation, manuscript editing, manuscript review.

Nelson J. Fonseca-Ruiz contributed in the concepts, design, definition of intellectual content, literature search, data analysis, manuscript editing, manuscript review.

Leo Anthony Celi contributed in the concepts, design, definition of intellectual content, literature search, manuscript preparation, manuscript editing, manuscript review.

John F. Duitama-Muñoz contributed in the concepts, design, definition of intellectual content, data analysis, manuscript preparation, manuscript editing, manuscript review.

The authors have no conflicts of interest to declare.