Acute respiratory failure (ARF) is a critical condition marked by the inability of the respiratory system to meet oxygenation or ventilation requirements. Approximately 60% of ARF patients require invasive mechanical ventilation (IMV), with a mortality rate of 30% and a leading cause of death in intensive care units (ICUs).1

Non-invasive mechanical ventilation (NIV) and high-flow nasal cannula (HFNC) are may be used to improve ARF and prevent endotracheal intubation (EI) and invasive mechanical ventilation.2 NIV provides support for ventilation and oxygenation most commonly through a face mask while HFNC mainly supports oxygenation by delivering up to 60 L/min of up to 100% humidified oxygen via the nasal route, may improve dead space CO2 washout and provide a low level of PEEP.3 Both modalities have become prevalent in ICUs.4 Recent guidelines indicate no significant difference in efficacy between HFNC and NIV for ARF treatment, although NIV is preferred in acute pulmonary edema and COPD.5–7 Both NIV and HFNC failure and subsequent delayed intubation can increase the mortality rate by up to 27–56%.8,9

The respiratory rate-oxygenation (ROX) index, defined as the ratio of oxygen saturation (SpO2) to the fraction of inspired oxygen (FiO2) divided by the respiratory rate (RR), can be easily and repetitively calculated from routinely collected data and was shown to have strong predictive performance for HFNC outcomes, and particularly for the need for intubation.10,11 Modified versions, such as the mROX (incorporating PaO2) and ROX-HR (normalization for heart rate), have also demonstrated good predictive performance in this setting.12,13 However, these indices have not been validated for predicting failure of NIV therapy.

This study aims to determine if the ROX index and its variations can predict the risk NIV therapy failure ICU patients using large public ICU databases containing highly granular routinely collected real world data.14

This report adheres to the STROBE reporting guidelines.15 Patient data was extracted exclusively from the publicly available AmsterdamUMCdb and MIMIC-IV databases and further ethics review was therefore not required.

Indices calculations

The ROX (I), mROX (II) and two newly emerged variations that add heart rate into calculations, ROX-HR (III) and mROX-HR (IV) were calculated as follows:

where SpO2 is mean blood oxygen saturation from 2-h windows, FiO2 is mean oxygen flow in a 2-h window, PaO2 is the mean arterial oxygen partial pressure in a 2-h window, RR mean respiratory rate for the 2-h window, and HR is the mean heart rate frequency for the specified 2-h window.

The AmsterdamUMCdb database contained 1075 patients who underwent episodes of NIV and we identified 3908 potential patients in MIMIC-IV database. After application of exclusion criteria, a total of 3344 patients were eligible for analysis, with 1344 (40.2%) patients designated as NIV failure within 24 h after an end of a NIV episode. The patient selection process is shown in Fig. 1.

Figure 1.

Flow diagram of patient selection process.

(0.47MB).

Patients baseline characteristics are presented in Table 1 and are stratified by source database in Appendix A Table 1. NIV failure patients had higher admission SOFA scores and higher CRP values. Patients were ventilated for relatively short periods of time, median 5.5 h (3.0, 11.0) for NIV success group and 5.0 h (2.9, 10.3) for NIV failure group, but without statistically significant difference between the groups.

Data completeness for calculating dynamic indices varied across time points and between databases. In the Amsterdam UMC ICU database, the highest availability was observed for the ROX index during the first 2 h of NIV (88.2%), while the mROX-HR index had the lowest availability at 10−12 h (16.5%). In the MIMIC-IV database, availability was generally lower, ranging from 26.6% for the ROX index at 0−2 h to 3.5% for the mROX-HR index at 10−12 h. In contrast, static demographic and baseline clinical data, such as age and gender, were nearly complete across both databases, reflecting the ease of capturing these infrequent, high-granularity variables compared to the time-sensitive, high-frequency ventilatory and hemodynamic data. Full details are provided in Appendix A.

The ROX index did not show statistically significant differences between the NIV success and failure groups across any of the time windows assessed. Similarly, the mROX index showed no significant differences in most time intervals, with the exception of a significant difference observed at the 2−4 h window (p = 0.015). After this period, the differences between the two groups for the mROX index were not statistically significant in the later time points. For the ROX-HR and mROX-HR indices, no statistically significant differences were found between the success and failure groups at any time window. Across all time intervals, both indices demonstrated comparable values between the groups. Additionally, patients in the NIV failure group had a significantly longer median length of hospital stay (100.8 h [54.4, 219.7]) compared to the success group (61.3 h [30.1, 135.8]) (p < 0.001). However, no significant difference in 28-day mortality was observed between the success group (32.2%) and the failure group (30.2%) (p = 0.245). The detailed results are presented in Table 2 and Fig. 2, with complete data available in Appendix A.

Figure 2.

Distribution of computed ROX index and its variations. Full dots denote outliers. In pairwise comparison with Mann–Whitney U test, only significant difference was found between mROX index at 2–4 h, p = 0.015.

(0.38MB).

The AUROC and precision-recall (PR) curve analyses for the ROX, mROX, ROX-HR, and mROX-HR indices showed consistently low to moderate predictive accuracy across different time windows. For the ROX index, the AUROC scores remained relatively low overall. The AUROC ranged from 0.515 at 0−2 h to 0.626 at 10−12 h. Precision-recall analysis showed somewhat consistent results, with the PR curve at 2−4 h showing the highest performance with a value of 0.552. The PR curve was lowest at 10−12 h with a value of 0.410. The mROX index demonstrated more stable AUROC values, peaking at 0.572 during the 4–6-h window. However, the precision-recall analysis indicated lower overall performance, with the PR curve ranging from 0.489 at 4−6 h to 0.387 at 10−12 h. Both the AUROC and PR curves displayed consistent performance across most time periods. For the ROX-HR index, the highest AUROC value was observed at 0−2 h (0.532), while the lowest was at 10−12 h (0.455). Precision-recall analysis revealed a higher PR curve value during the 4–6-h window (0.594), while lower values were observed at other time points, such as 0.375 at 8−10 h. Finally, the mROX-HR index had the lowest overall performance in both AUROC and PR curve analyses. The AUROC values remained below 0.526 throughout, with the highest observed for 0−2 h. Precision-recall analysis mirrored these results, with the highest PR curve value reaching 0.484 for 0−2 h and dropping as low as 0.333 at 6−8 h. Results are summarized in Figs. 3 and 4.

Figure 3.

ROC curves and AUROC scores for respiratory indices predicting non-invasive ventilation success or failure across successive time intervals.

(0.63MB).

Figure 4.

Precision-recall analysis of ROX, mROX, ROX-HR and mROX-HR indices at various time points.

(0.69MB).

To the best of our knowledge, this is the first report on the performance of the ROX index and its variations to predict the risk of NIV therapy failure in ICU patients using real world data from two large publicly available ICU databases. We found that ROX index and its variations mROX, ROX-HR, and mROX-HR do not differentiate between NIV therapy success or failure, but their predictive performance must be considered moderate at best with only ROX at 10−12 h after start of NIV ventilation reaching a clinically acceptable value (AUROC = 0.626).

The original ROX index has previously been shown to have good predictive performance for intubation within 24 h for patients on HFNC due to community-acquired pneumonia or SARS-CoV-2 ARF.19,20 Similarly, a recent study has demonstrated the ROX index as a good predictor for invasive IMV in patients receiving HFNC due to pneumonia, with an area under the curve of receiver operating characteristics (AUROC) of 0.77 after 24 h.21

Data on the performance of the original ROX index to predict NIV failure is limited. For patients with de novo acute respiratory failure, the ROX index was used to predict NIV failure at 12 h of ventilation yielding an AUROC of 0.74, comparable to our findings and also moderately discriminating at best.22 Similar to our study, patients with NIV failure had higher SOFA scores (median 4.0 vs. 5.0, p < 0.001) compared to those with NIV success. Our study included a larger sample size (3344 vs. 1286 patients) and had a lower percentage of NIV failure (59.8% vs. 40.1%). A recent retrospective study on the ROX index's predictive value for NIV failure within 48 h after initiation in COVID-19 patients showed an AUROC of 0.86, indicating good predictive performance.23 However, this study involved only 72 patients, with 50 requiring mechanical ventilation and NIV failure of only 26%, with calculated ROX index values while patients were breathing room air before starting ventilatory support.

Variations of the original ROX index include dividing ROX by heart rate to yield the ROX-HR index and substituting SpO2 with PaO2 to yield the mROX and mROX-HR indices. It has been found that the mROX index has AUROC values over 0.8 for predicting HFNC failure two hours after initiation with similar predictive performance reported for ROX-HR and mROX-HR.24 For COVID-19 patients, the mROX had an AUROC of 0.956 after two hours, compared to 0.894 for the ROX index.25 To our knowledge, none of these modified versions of the ROX index have previously been studied as potential tools for predicting the failure of NIV/CPAP ventilation. In our study, the highest AUROC values were 0.626 for ROX at 10−12 h after NIV initiation, with none of the other indices in any of the timepoints reaching at least moderate predicting value.

Reasons for worse performance of ROX index and derivatives for NIV failure compared to HFNC may be versatile. HFNC is easier to tolerate, may be chosen for less severe patients and might deliver lower but more constant flows. ICU ventilators are deigned to compensate for high flows and system leaks for up to 120 L/min but demands in respiratory distress syndrome can go up to 200 L/min of peak inspiratory flow rate if the leak in the circuit is present, affecting the final FiO2 delivered.26,27 Findings based on research on models and ideally breathing subjects suggest that increasing leaks lead to more frequent auto-triggering and miss-triggering, causing constant tidal volume delivery problems.28 Integrating flow considerations into the formula could provide a more accurate assessment of patient status in scenarios with variable leak and flow conditions.

This study comes with limitations. These include lack of data completeness, especially for longer durations of NIV as well as a surprisingly low number of patients who had complete data for NIV ventilation for 10 h or more. Data from Amsterdam UMC has good density of granular data for this type of research, but MIMIC-IV, as suggested by previous research, suffered from lack of granularity.29 Another limitation may be that, due to relatively rare data on PaO2 compared to FiO2, oxygen saturation and respiratory rate available in the database, values had to be extracted in such way that the nearest FiO2 from the ventilator or estimated FiO2, not more than 1 h apart, had to be used, a design that would certainly be improved if the study was prospective. Additionally, in some cases databases designate modes of respiratory support under a combined CPAP/NIV label, making it impossible to assess whether the effect of the indices differs between CPAP and NIV specifically.

In conclusion, neither ROX index nor any of the modified indices demonstrated sufficient predictive value to be useful tools for predicting NIV failure and cannot currently be recommend for clinical use. Predictive power should be tested in addition to other tools used for respiratory monitoring.

LL: Conceptualization, Methodology, Software, Data curation, Writing, Visualization, Formal Analysis. PE: Supervision, Writing – Rewriting & Editing, Methodology, Resources. TR, NK, MV: Writing – Rewriting & Editing.

No generative AI system has been used in the writing or editing of this manuscript.

None.