To validate Trauma and Injury Severity Score (TRISS) methodology as an auditing tool in the Spanish ICU Trauma Registry (RETRAUCI).
DesignA prospective, multicenter registry evaluation was carried out.
SettingThirteen Spanish Intensive Care Units (ICUs).
PatientsIndividuals with traumatic disease and available data admitted to the participating ICUs.
InterventionsPredicted mortality using TRISS methodology was compared with that observed in the pilot phase of the RETRAUCI from November 2012 to January 2015. Discrimination was evaluated using receiver operating characteristic (ROC) curves and the corresponding areas under the curves (AUCs) (95% CI), with calibration using the Hosmer–Lemeshow (HL) goodness-of-fit test. A value of p<0.05 was considered significant.
Main variables of interestPredicted and observed mortality.
ResultsA total of 1405 patients were analyzed. The observed mortality rate was 18% (253 patients), while the predicted mortality rate was 16.9%. The area under the ROC curve was 0.889 (95% CI: 0.867–0.911). Patients with blunt trauma (n=1305) had an area under the ROC curve of 0.887 (95% CI: 0.864–0.910), and those with penetrating trauma (n=100) presented an area under the curve of 0.919 (95% CI: 0.859–0.979). In the global sample, the HL test yielded a value of 25.38 (p=0.001): 27.35 (p<0.0001) in blunt trauma and 5.91 (p=0.658) in penetrating trauma. TRISS methodology underestimated mortality in patients with low predicted mortality and overestimated mortality in patients with high predicted mortality.
ConclusionsTRISS methodology in the evaluation of severe trauma in Spanish ICUs showed good discrimination, with inadequate calibration – particularly in blunt trauma.
Evaluar el Trauma and Injury Severity Score (TRISS) como instrumento de auditoría en el Registro Español de Trauma en UCI.
DiseñoEvaluación prospectiva de un registro multicéntrico.
ÁmbitoTrece UCI españolas.
PacientesIndividuos con enfermedad traumática y datos completos ingresados en las UCI participantes.
IntervencionesComparamos la mortalidad predicha por el TRISS con la observada en la fase piloto del Registro Español de Trauma en UCI desde noviembre de 2012 hasta enero de 2015. La discriminación se evaluó mediante curvas receiver operating characteristic y el valor bajo su área (IC 95%), y la calibración, mediante el test de bondad de ajuste de Hosmer-Lemeshow. Un valor de p<0,05 se consideró significativo.
Principales variables de interésMortalidad observada y predicha.
ResultadosAnalizamos 1.405 pacientes. La mortalidad observada fue del 18% (253 pacientes), mientras que la predicha fue del 16,9%. El área bajo la curva receiver operating characteristic fue de 0,889 (IC 95% 0,867-0,911). Los pacientes con trauma cerrado (n=1.305) presentaron un área bajo la curva receiver operating characteristic de 0,887 (IC 95% 0,864-0,910), y aquellos con traumatismo penetrante (n=100), de 0,919 (IC 95% 0,859-0,979). En la muestra global, el test de Hosmer-Lemeshow mostró un valor de 25,38 (p=0,001), siendo de 27,35 (p<0,0001) en trauma cerrado y de 5,91 (p=0,658) en trauma penetrante. La metodología TRISS infraestimó la mortalidad en los pacientes con mortalidad predicha baja y la sobreestimó en pacientes con mortalidad predicha elevada.
ConclusionesLa aplicación de la metodología TRISS en el trauma grave ingresado en las UCI españolas mostró buenos niveles de discriminación y una calibración inadecuada, especialmente en el traumatismo cerrado.
Trauma registries constitute an useful tool for monitoring trauma patient care, since they accurately reflect management and care in different settings.1,2 This allows not only improvement of individual treatment but also reorganization of the general care profile and logistics applied in the management of severe trauma patients, as well as comparisons among different registries for benchmarking purposes.1–4
The outcome of trauma patients is mainly determined by the initial severity of the physiological and anatomical injuries secondary to trauma itself. Severity of injury can be evaluated by severity scales obtained from the analysis of large cohorts of patients.5 Of all the prognostic scores used in trauma patients, the Trauma and Injury Severity Score (TRISS) is currently the most widely used tool, and is regarded as the standard method. In this regard, TRISS methodology determines the probability of survival based on a logistic regression model that includes anatomical evaluation by means of the Injury Severity Score (ISS), physiological evaluations using the Revised Trauma Score (RTS), patient age and type of trauma (blunt or penetrating).4 The following equation is applied:
Coefficients b0–b3 were derived from the logistic regression analysis of the American database MTOS (US Major Trauma Outcome Study). This was not a population-based database, participation was voluntary, and it involved hospitals with a special interest in trauma.
The pilot phase of the Spanish Intensive Care Unit Trauma Registry (RETRAUCI) has been recently conducted in 13 Intensive Care Units (ICUs).6 Our objective was to validate TRISS methodology almost 30 years later as an auditing tool for mortality prediction in the Spanish ICU Trauma Registry, taking into consideration the mechanisms of injury (blunt or penetrating).
MethodsThe pilot phase of the RETRAUCI was conducted from 23 November 2012 to 31 January 2015. Thirteen ICUs distributed throughout Spain collected data. The RETRAUCI is endorsed by the Trauma and Neurointensive Care Working Group of the Spanish Society of Intensive Care Medicine (SEMICYUC). Ethics Committee approval for the registry was obtained. No specific interventions were required for this study.
PatientsWe studied all patients admitted to the participating ICUs during the pilot phase of the RETRAUCI due to traumatic disease. In all cases, data on epidemiology, acute management, resource utilization and outcome were recorded. Patients were followed-up on until hospital discharge for outcome as a dichotomous variable (alive or dead).
The following exclusion criteria were applied:
- -
Missing data for calculating RTS or ISS.
- -
Outcome at hospital discharge not known for any reason.
Data used for calculating RTS (respiratory rate, systolic blood pressure and Glasgow coma score) were obtained from first medical attention before initiating resuscitation and/or mechanical ventilation.
Data used for calculating the ISS were prospectively collected by the intensivist in charge of the patient after ICU admission, based on the Abbreviated Injury Scale (updated in 2008).
Statistical analysisQuantitative data are reported as means (standard deviation) (SD) and qualitative data as absolute frequencies and percentages. Probability of survival was calculated according to TRISS methodology5,7 and secondarily, predicted mortality was calculated as follows: (100 – predicted probability of survival). To evaluate the validity of the model, we studied discrimination and calibration in the whole sample and distributed according to blunt or penetrating mechanisms of injury. Discrimination refers to the ability to distinguish between patients who die and those that survive. Accordingly, if the model predicts a mortality rate of 20%, discrimination is perfect if the observed mortality is 20%. It can be evaluated using receiver operating characteristic (ROC) curves and the area under the curves (AUCs) (95% confidence interval, 95% CI). The greater the area, the better the discrimination. Sensitivity (S), specificity (Sp), and the positive (PPV) and negative predictive values (NPV) were recorded in each case.
The calibration of a prognostic model evaluates the concordance between the probability observed in the sample and the probability predicted by the model, describing how the prognostic scale works over wide ranges of predicted mortality. It is evaluated using the Hosmer–Lemeshow (HL) goodness-of-fit test, in patients with low (<10%), intermediate (10–50%) and high predicted mortality rates (>50%). A probability of close to 1 represents better adjustment.8
Statistical significance was considered for p<0.05. The SPSS® version 20 statistical package (IBM Corporation 2011) was used throughout.
ResultsA total of 2242 patients formed the global cohort of the pilot phase of the RETRAUCI. Patient dropout from the sample was due to different reasons, the most important being inability to determine hospital outcome in 21.1% of the cases, due to patient transfer to the corresponding reference hospital. In most cases, transfer was done to another country, making follow-up impossible. The flowchart in Fig. 1 summarizes the final sample of 1405 patients included in the study.
The mean patient age was 46.7 (19.4) years, and 80.3% were males. The mean ISS score was 21.3 (12.1). A total of 1305 patients suffered blunt trauma (92.9%) as the main mechanism of injury, while the remaining 100 patients (7.1%) presented penetrating trauma. Table 1 shows the data referred to patient epidemiology, acute management, resource utilization and main outcome.
Epidemiological and clinical data of the 1405 patients included in the study.
Variable | Patients (N=1405) Number (%) | |
---|---|---|
Trauma mechanism | RTA | 571 (40.7%) |
Fall | 394 (28.1%) | |
Aggression | 105 (7.5%) | |
Occupational accident | 100 (7.1%) | |
Self-injury | 84 (6%) | |
Sports-related | 81 (5.8%) | |
Others | 69 (4.9%) | |
Out-of-hospital medical support | 1240 (88.8%) | |
ISS | <15 | 429 (30.5%) |
15–25 | 600 (42.7%) | |
26–50 | 342 (24.3%) | |
>50 | 34 (2.4%) | |
Hemodynamically stable-admission | 878 (66%) | |
ICP monitoring | 233 (18.1%) | |
Blood transfusion 24h | 351 (25%) | |
Mechanical ventilation | 806 (66.2%) | |
Mechanical ventilation (days) | 6.7 (8.6) | |
MOF | 131 (10%) | |
ICU stay (days) | 8.1 (9.6) | |
Post-ICU stay (days) | 14.8 (19.9) | |
Global mortality | 253 (18%) |
RTA, road traffic accident; ISS, Injury Severity Score; ICP, Intracranial pressure; MOF, Multiorgan failure; ICU, Intensive Care Unit.
The observed mortality rate (including ICU and post-ICU stay) was 18% (253 patients), with a predicted mortality rate of 16.9%. In patients with blunt trauma, the observed and predicted mortality rates were 18.5% and 17.1%, respectively. In penetrating trauma, the observed and predicted mortality rates were 12% and 14.2%, respectively. Table 2 shows the distribution of patients who died in groups with low (<10%), intermediate (10–50%) and high predicted mortality (>50%).
The global sample of 1405 patients presented an area under the ROC curve of 0.889 (95% CI: 0.867–0.911), with S=50.9%, Sp=96.2%, PPV=74.6% and NPV=89.9%. Patients with blunt trauma (Fig. 2) presented an area under the ROC curve of 0.887 (95% CI: 0.864–0.910), with S=50.6%, Sp=96.2%, PPV=75.3% and NPV=89.6%, while patients with penetrating trauma (Fig. 2) presented an area under the ROC curve of 0.919 (95% CI: 0.859–0.979), with S=58.3%, Sp=95.5%, PPV=63.6% and NPV=94.4%.
The results of the Hosmer–Lemeshow (HL) goodness-of-fit test, in both in the total cohorts of patients of patients and distributed according to the mechanism of trauma (blunt or penetrating) are shown in Table 3. The correlation between predicted and observed mortality is shown in Fig. 3.
Hosmer–Lemeshow (HL) goodness-of-fit test.
Predicted mortality deciles (%) | N | Observed survivors | Predicted survivors | Observed dead | Predicted dead | H–L df p-value |
---|---|---|---|---|---|---|
Total sample (N=1405) | ||||||
0–0.8 | 175 | 174 | 173.9 | 1 | 1.1 | |
0.9–1.2 | 157 | 157 | 155.4 | 0 | 1.6 | |
1.3–2.2 | 152 | 146 | 149.2 | 6 | 2.8 | |
2.3–3.5 | 136 | 128 | 132.1 | 8 | 3.9 | |
3.6–5.7 | 129 | 124 | 122.9 | 5 | 6.1 | |
5.8–9.1 | 134 | 120 | 124.5 | 14 | 9.5 | |
9.2–15 | 130 | 105 | 115.4 | 25 | 14.6 | |
15.1–31.7 | 140 | 108 | 109.3 | 32 | 30.7 | 25.38 8 0.001 |
31.8–63.6 | 123 | 64 | 66.3 | 59 | 56.7 | |
63.7–100 | 129 | 26 | 20.2 | 103 | 108.8 | |
Blunt trauma (N=1305) | ||||||
0.3–0.7 | 123 | 122 | 122.1 | 1 | 09 | |
0.8–1 | 116 | 116 | 114.9 | 0 | 1.1 | |
1.1–1.7 | 143 | 141 | 141 | 2 | 2 | |
1.8–3 | 134 | 127 | 130.8 | 7 | 3.2 | |
3.1–5 | 132 | 124 | 126.7 | 8 | 5.3 | |
5.3–7.8 | 134 | 122 | 125.5 | 12 | 8.5 | |
7.9–13 | 132 | 107 | 118.4 | 25 | 13.6 | |
13.1–27.4 | 132 | 108 | 98.3 | 24 | 33.7 | 27.35 8 <0.001 |
27.5–63.2 | 134 | 71 | 76.7 | 63 | 57.3 | |
63.3–100 | 125 | 26 | 18.8 | 99 | 106.2 | |
Penetrating trauma (N=100) | ||||||
0.3–0.7 | 10 | 10 | 9.94 | 0 | 0.06 | |
0.8–0.9 | 11 | 11 | 10.9 | 0 | 0.1 | |
1–1.3 | 10 | 10 | 9.9 | 0 | 0.1 | |
1.4–2.1 | 10 | 10 | 9.8 | 0 | 0.2 | |
2.2–2.6 | 10 | 10 | 9.7 | 0 | 0.3 | |
2.7–4.7 | 9 | 9 | 8.6 | 0 | 0.4 | |
4.8–7 | 11 | 9 | 10.3 | 2 | 0.7 | |
7.1–31 | 9 | 8 | 8.1 | 1 | 0.9 | 5.91 8 0.66 |
31.1–54.8 | 10 | 8 | 6.7 | 2 | 3.3 | |
54.9–100 | 10 | 3 | 1.6 | 7 | 8.4 |
N=number of patients; df, degrees of freedom; H-L, Hosmer–Lemeshow goodness-of-fit test.
Our study shows that TRISS methodology applied in the sample of patients included in the pilot phase of the RETRAUCI presented good levels of discrimination with inadequate calibration, especially in patients with blunt trauma. Penetrating trauma showed better discrimination and good calibration. Altogether, these results suggest that newly calibrated (b coefficient) scales are necessary in our setting.
Our sample of patients offers an initial picture of patients with severe trauma admitted to the ICUs of our setting, taking into account the severity of injury, the care provided, length of stay and mortality. These Units represent level I and II centers. Such patients usually present high ISS values, important resource utilization, and high mortality. The TRISS methodology is based on the degree of anatomical injury (ISS), physiological response (RTS) and functional reserve (age). It was first developed in the 1980s through several logistic regression models5,9 with different b coefficients considering blunt or penetrating injuries. Several updates have been made since then.
When applied to our patients, TRISS showed good discrimination with inadequate calibration – a fact that limits the use of this prognostic model. This observation is consistent with other studies, and in general, with other prognostic scores in the ICU setting, where the main shortcoming corresponds to inadequate calibration despite good discrimination.10 It must be noted that in our sample of patients, those with penetrating injury showed good calibration. This fact is consistent with previous studies differentiating between blunt and penetrating trauma. In the latter type of trauma, discrimination and calibration is better, perhaps due to lesser improvement in their specific care.11,12
Poor calibration and discrimination does not necessarily refer to the quality of the care provided but rather to incorrect application of the model to a population with specific characteristics.13 In addition, TRISS is considered to present lower sensitivity for blunt trauma, since it underestimates brain injury; does not consider multiple injuries in the same anatomical area; and does not consider age on an individual basis. In our sample of patients, on taking into account the different mortality groups, TRISS underestimated mortality when the predicted mortality was <60% and overestimated it when the predicted mortality was >60% (Fig. 3).
Mortality prediction according to TRISS has therefore been questioned14,15: its clinical application has shown opposite results,16–18 especially when used in non-MTOS patients.19,20 The best way to increase its predictive value is to use local correction factors to adjust for b coefficients.20–22 The results of our study confirm that newly developed b coefficients are needed for trauma patients admitted to Spanish ICUs.
New scores have been developed in an attempt to improve the predictive value of TRISS. Some of them are the ASCOT (A Severity Characterization Of Trauma), which includes gender, 5 age categories and different anatomic scales; the ICISS (International Classification of Diseases Ninth revision based injury severity score); the NISS (New Injury Severity Score); the RISC (Revised Injury Severity Classification score); or the pediatric BIG score. However, although they have slightly improved the predictive ability of TRISS methodology, the latter remains the most widely used tool in clinical practice.12,20,23–25
Our study has a number of limitations – some attributable to the TRISS model itself, and other specific of our sample. The most relevant are (a) the limited number of patients for this kind of analysis despite the multicenter nature of the study. This corresponds to the pilot phase of the RETRAUCI. With a growing number of centers recruiting patients, we expect to solve this issue in the future; (b) up to 21% of the patients were lost for hospital outcome evaluation. This was due to the large number of patients from different countries that are admitted to our ICUs and are subsequently transferred to their reference hospitals at home, thereby making follow-up impossible.
In sum, TRISS methodology in the Spanish RETRAUCI showed good levels of discrimination, with inadequate calibration, especially in blunt trauma. Penetrating trauma showed better discrimination and good calibration. Altogether, these results suggest that newly calibrated (b coefficient) scales are necessary in our setting.
CollaboratorsCarlos García-Fuentes, Luis Terceros-Almanza (Hospital Universitario 12 Octubre), Iker García-Sáez, Ruth Salaberria-Udabe (Hospital Universitario de Donostia), Cecilia Carbayo-Górriz (Complejo Hospitalario de Torrecárdenas), Carmen Corcobado-Márquez (Hospital General Universitario de Ciudad Real), Javier Trujillano-Cabello (Hospital Universitari Arnau de Vilanova), Eduardo Miñambres-García (Hospital Universitario Marqués de Valdecilla), Amanda Lesmes-González (Hospital Universitario de Getafe).
FundingRETRAUCI received funding from the Fundación Mutua Madrileña during three years for the development of a web-based database, granted to the principal investigator (Dr. Chico-Fernández, AP117892013), on behalf of the Trauma and Neurointensive Care Working Group of the SEMICYUC.
Conflicts of interestAll authors listed in the study declare that they have no conflicts of interest.
The authors are indebted to Vicente Gómez-Tello (Hospital Moncloa, Madrid. Working Groups Coordinator, SEMICYUC), Andrés J. Chacón (Project Manager, Xferic) and Miguel Ferrero-Fernández (Department of Informatics, SEMICYUC) for their invaluable help in developing the registry and the web-based tool.