We conducted a descriptive, observational, metric study in a polyvalent ICU of a teaching hospital. The Guidelines for Reporting Reliability and Agreement Studies (GRRAS) checklist was followed.38

Patients >18 years of age with expected ICU stays > 48 h who consented to participate and were admitted from January 2020 through June 2023 were included. Patients with suspected imminent death or COVID-19 were excluded.

Based on the COSMIN design checklist for outcome measurement instruments,39 at least, 100 pairs of evaluations by 2 independent observers would be required to analyze the inter-observer reliability. Considering a 10% dropout rate, a minimum of 111 evaluation pairs was needed.

Upon admission, frailty was assessed using the Clinical Frailty Scale-España (CFS-Es)13 and the FRAIL-España (FRAIL-Es).14 Other recorded variables included sex, age, comorbidities (Charlson Comorbidity Index40), and severity level (Simplified Acute Physiology Score 3 [SAPS 3]41).

All participants in the study were assessed for frailty based on their condition in the month prior to hospital admission, using the CFS-Es and FRAIL-Es. The evaluations were conducted in person by 3 intensive care nurses (N1, N2, and N3), with more than 20 years of ICU experience each, and 2 intensivists (I1 and I2) with more than 10 years of specialty experience. Assessments were performed with patients directly or their relatives when patients were not capable of communicating. Not all patients could be evaluated multiple times or by all evaluators. One intensivist (I1) conducted only 1 single evaluation (always with the patient), while the remaining professionals conducted up to 2 evaluations (with relatives and/or the patient), at least, 72 h after the first evaluation. All evaluations were conducted independently. If multiple evaluators coincided during a patient visit, all could ask questions, but the individual frailty scores assigned were always blinded to the other evaluators. Upon patient discharge, the lead investigator had access to each individual evaluation from each evaluator. Although the professionals conducting the frailty evaluations were familiar with the scales, the use of such scales was not part of their routine clinical practice.

The evaluations by N1 were considered the reference among nurses because she was the study lead investigator and conducted the initial interview with the patients or their relatives, which also included questions about dependency and quality of life. Additionally, she was the only nurse to assess all patients included in the study. Evaluations by I2 were considered the reference among intensivists, as he was the only one who conducted 2 evaluations per patient. Comparisons across nurses were established between N1 and N3 due to the smaller number of assessments conducted by N2.

Quantitative variables are expressed as median and interquartile range [Q1–Q3], following non-parametric testing (Shapiro-Wilk). Qualitative variables are expressed as absolute (n) and relative (%) frequencies. Group median comparisons were performed using the Mann–Whitney U test.

Concordance was evaluated both within the same professional and among 2 and 3 professionals for the CFS-Es and FRAIL-Es scales using the ICC, through the comparison of variances (ANOVA) with repeated measures,42 when the scales were considered as quantitative variables. Cohen's Kappa (K)43 coefficient was used for comparisons between 2 observers, and Fleiss' Kappa44 was used to assess concordance among 3 observers. Concordance was classified as slight if ≤ 0.20, fair if 0.21–0.40, moderate if between 0.41 and 0.60, substantial if between 0.61 and 0.80, and almost perfect if > 0.80.

Frailty assessments using the CFS-Es stratified patients into 3 options: option A categorized patients into 3 groups: non-frail (CFS-Es = 1–3); vulnerable or with very mild frailty (CFS-Es = 4); frail (CFS-Es = 5–9). Options B and C categorized patients into 2 groups (non-frail and frail): In option B, patients at level 4 were considered non-frail (frail = CFS-Es 5–9). In option C, patients at level 4 were considered frail (frail = CFS-Es 4–9). These stratifications were introduced following a change in nomenclature for level 4 in 2020,9 when it shifted from "vulnerable" (non-frail) to "with very mild frailty" (frail). Assessments using the FRAIL-Es stratified patients into 3 groups (option A): non-frail (FRAIL-Es = 0); pre-frail (FRAIL-Es = 1–2); or frail (FRAIL-Es = 3–5), or 2 groups (Option B): FRAIL-Es scores from 0 to 2 categorized patients as non-frail (Table 1).

Since the FRAIL-Es includes dichotomous items, internal consistency was estimated using the Kuder-Richardson Formula 20.45 Reliability was categorized as very low (<0.20), low (0.21−0.40), moderate (0.41−0.60), high (0.61−0.80), and very high: (>0.80).46 The correlation of each item with the overall scale score was evaluated using the Spearman correlation coefficient and was categorized as null (<0.10), weak (0.10−0.29), moderate (0.30−0.50), and strong correlation (>0.50).47 The internal consistency of the FRAIL-Es was calculated using all evaluations conducted by the 5 professionals.

All values are presented with 95% confidence intervals (CI95%) and p-values, with statistical significance set at p < 0.05. Statistical analyses were performed using IBM SPSS Statistics for Windows (version 29.0, IBM Corp., Armonk, NY, United States).

The study protocol was reviewed and approved by the hospital Research Ethics Committee (CEIm2019/42). Consent was obtained from patients or their closest relatives when personal consent was not possible. Data confidentiality was maintained by assigning alphanumeric codes known only to the principal investigator.

The ICC demonstrated near-perfect agreement between first and second evaluations by each professional, with values >0.80 for the CFS-Es and >0.90 for the FRAIL-Es (Table 3).

When stratifying the scales, agreement levels ranged from substantial to near-perfect for both the CFS-Es and FRAIL-Es, depending on the stratification options. No significant differences in reliability were seen between nurses and intensivists. The FRAIL-Es dimensions with the lowest agreement were fatigue and ambulation, while comorbidities showed the highest agreement (Table 3).

Agreement among different professionals, measured by ICC (95%CI), was nearly perfect for both CFS-Es and FRAIL-Es. The lowest agreement was seen in FRAIL-Es evaluations among intensivists [ICC (95%CI), 0.763 (0.645–0.842); p < 0.001] (Table 4).

Stratified scale analysis revealed moderate to substantial agreement for both the CFS-Es and FRAIL-Es, depending on stratification. The highest agreement for CFS-Es occurred when patients were categorized into 2 strata (non-frail and frail), considering patients at level 4 as frail (Option C). Intensivists showed lower agreement vs nurses. Also, agreement was lower between intensivists and nurses. FRAIL-Es dimensions with the lowest agreement among professionals were weight loss and fatigue, while comorbidities showed the highest agreement (Table 4).

The FRAIL-Es showed high reliability measured by Kuder-Richardson Formula 20 (95%CI), 0.643 (0.608–0.677). Strong correlations were observed between 4 different dimensions (Fatigue, Resistance, Ambulation, and Involuntary Weight Loss) and the global scale score, while the comorbidities dimension showed weak correlation (Table 5).

The CFS-Es intra-observer reliability is consistent with the results reported by Abraham et al. in the validation of the French version implemented by nurses [ICC (95%CI), 0.87 (0.76–0.93)] and intensivists [ICC (95%CI), 0.86 (0.72–0.93)]. Other validated versions in non-critical patients also show excellent intra-observer reliability, such as the Portuguese30 [ICC (95%CI), 0.999 (0.998–0.999)] and Greek versions31 [ICC (95%CI), 0.89 (0.85–0.92)]. The Brazilian version29 reported lower reliability (ICC, 0.641). These findings confirm that the CFS-Es has very good intra-observer reliability, whether implemented by nurses or intensivists.

Furthermore, the FRAIL-Es also shows very high intra-observer reliability, outperforming the Mexican (ICC, 0.82) and Chinese (ICC, 0.708) versions. The comorbidities dimension achieved 100% agreement, likely due to its objective assessment based on patient health records, whereas other dimensions relied on information from patients or families.

The reliability of the CFS-Es across different observers was similar to the one reported by Vrettos et al.31 in their validation study of the Greek version in non-critical hospitalized elderly patients [ICC (95%CI), 0.87(0.82−0.90)]. On the other hand, the reliability data of the CFS-Es exceed that reported by Abraham et al.22 in the validation of the French version among intensivists [ICC (95%CI), 0.76(0.57−0.87)] intensive care nurses [ICC (95%CI), 0.76(0.57−0.87)] or between intensivists and nurses [ICC (95%CI), 0.75(0.56−0.87) and 0.73(0.52−0.85), respectively, as well as the reliability data reported by Rodrigues et al.29 (ICC, 0.350) in the validation of the Brazilian version among volunteers. In their validation of the Danish version, Nissen et al.26 achieved higher reliability among intensivists [ICC (95%CI), 0.90(0.82−0.96)], but this validation was not conducted with patient or family interviews. Instead, it involved the evaluation of written case reports, thereby excluding potential variability in patient and family responses. Even so, their reliability data are very close to those reported for the CFS-Es.

Although we did not find any studies evaluating the reliability of the FRAIL scale in critical patients, 2 studies that evaluated it among non-hospitalized adults report reliability scores lower than those of the FRAIL-Es (ICC, 0.82 in the Mexican version35 and ICC, 0.708 in the Chinese version3).4

Regarding agreement among groups, 5 studies implemented in critical patients evaluate the reliability of the CFS.23–25,28,32 In the study by Flaatten et al.,32 intensivists and intensive care nurses assessed the frailty of elderly patients (<80 years) and categorized them into 3 strata (CFS = 1–3 not frail, CFS = 4 vulnerable, and CFS = 5–9 frail). The reliability obtained among intensivists was higher than that of the CFS-Es [K(95%CI, 0.80 (0.77−0.84)] as was the reliability among nurses [K(95%CI, 0.77(0.71−0.83)], and between intensivists and nurses [K(95%CI, 0.77(0.71−0.83) and 0.80(0.77−0.84), respectively]. This superiority may be related to the fact that it was a multicenter study with 3920 patients from 22 European countries that obtained a total of 1923 pairs of evaluations. Surkan et al.28 implemented the CFS (the original English version) in 158 ICU patients older than 18 years. The reliability between an intensivist and a geriatric resident or geriatrician was much lower than that obtained for the CFS-Es [K (95%CI, 0.32 (0.17–0.46) and 0.29 (−0.11 to 0.69), respectively].

In the study by Pugh et al.,25 the CFS (original in English) was also implemented by intensivists and intensive care nurses. However, in this case, patients were categorized into 2 groups (CFS = 1–4 and CFS = 5–9). The reliability obtained was more similar to that of the CFS-Es [K(95%CI, intensivist-nurse 0.59 (0.44–0.75), nurse-nurse 0.63 (0.45–0.82)], except among intensivists, in whom it was higher [K (95%CI, 0.70 (0.67–0.80)]. However, the agreement percentage among intensivists was 63% vs 82% for the CFS-Es. Similarly, Hope et al.,24 who analyzed variability among researchers, patients, or relatives, obtained reliability scores among the 3 groups (CFS = 1–3, CFS = 4, and CFS = 5–9; K (95%CI, 0.43 (0.42–0.46)] similar to the overall reliability of the CFS-Es and slightly higher [K (95%CI, 0.62 (0.53 to 0.70)] when analyzing variability by categorizing patients into 2 groups (CFS = 1–4 and CFS = 5–9). In a retrospective study, Darvall et al.23 reported a Kappa coefficient of 0.67 and an agreement percentage of 45% between the observations made by a researcher and a resident intensivist, which is a Kappa coefficient slightly higher than the CFS-Es with a lower agreement percentage. Moreover, it is important to note that the evaluations from study by Darvall et al.23 were conducted after reviewing the patients’ health records, excluding potential variability from patient or family interviews.

Finally, the inter-observer reliability by groups for the CFS-Es is lower than that reported for the Turkish27 (K, 0.811) and Chinese33 (K, 0.60) versions, both implemented in ambulatory patients older than 65 years.

We did not find any studies evaluating inter-observer agreement for the FRAIL scale or its different versions. The overall data for the FRAIL-Es are moderate, whether analyzing agreement between 2 (non-frail and frail) or 3 groups (non-frail, pre-frail, and frail). The evaluations conducted by intensivists yielded the lowest scores, likely due to moderate agreement in the illness and weight loss dimensions. The Kappa coefficient of 0.492 among intensivists in the illness dimension, was undoubtedly influenced by the sample, as the agreement among intensivists was 98%, and the 2 evaluations were based on the same health record. Unintentional weight loss showed fair agreement among intensivists and moderate or substantial agreement in other comparisons (intensivist vs nurse and nurse vs nurse, respectively), possibly due to the difficulty in quantifying this dimension. Some patients were never weighed and unaware of their weight change, though repeated questioning could lead to reflection and greater clarity in the second evaluation.

The internal consistency of FRAIL-Es, measured using the Kuder-Richardson Formula 20, is higher than that observed by Dong et al.34 in the Chinese version of the FRAIL (KR-20 = 0.485) and by Aprahamian et al.36,37 in the Brazilian Portuguese version (KR-20 (95%CI), 0.447 (0.290–0.605) and KR-20 = 0.53). Although a high internal consistency was achieved, it may not have been higher due to the low number of items (5) included in the scale.46 The correlation between the items of the scale and the overall assessment was strong, except for the comorbidities item. This issue has also been reported by Rosas-Carrasco et al.35 in the Mexican Spanish version and by Susanto et al.,48 who obtained lower correlations than those of FRAIL-Es. Rosas-Carrasco et al.35 suggested that the reason for this low correlation may be that comorbidities are not as closely related to the development of frailty as the other dimensions of the scale are. However, in our cohort, we observed that frail patients (FRAIL-Es = 3–5) had a significantly higher Charlson Index vs non-frail patients (FRAIL-Es = 0–2). Certainly, comorbidities are related to frailty, though possibly to a lesser degree than the other dimensions evaluated. Alternatively, this may be related to representativeness, as only 4% of patients exhibited the comorbidities dimension.

The primary strength of this study is evaluating whether the scales are equally reliable when implemented by intensive care nurses or intensivists. Frailty assessment should be interprofessional, as a patient's frailty level may impact nursing care and/or medical treatment.

Disagreements when evaluating the same patient (either between different evaluators or by the same evaluator at different times) could have been influenced by varying responses from patients and families. It was not always possible to conduct all evaluations with the patient or their relatives. Additionally, responses could be influenced by social desirability bias, where participants only give responses they believe to be appropriate rather than truthful. When multiple relatives were present, efforts were made to include everyone in the responses to reach a consensus, but this was not always feasible. Obtaining an objective assessment is challenging, both with a scale using closed questions (like FRAIL-Es) and with open-ended scales dependent on the professional's expertise (like CFS-Es).

Furthermore, studies evaluating reliability in critical patients are scarce, with some focusing on very specific populations (>60 or >80 years) and implemented by different professionals, making comparisons with the present study difficult.

Another limitation could be the lack of prior training for professionals implementing the scales. Although they were familiar with the scales, they did not use them routinely, and the learning curve may have influenced their reliability.49 In this regard, comparisons with other studies were not possible, as such data were not reported.

It could also be considered a limitation the fact that not all evaluators were able to assess patient frailty at 2 different timeframes. While desirable, the patients’ clinical and/or care conditions did not always allow this.

After establishing the reliability of the scales when implemented in critical patients, it would be interesting to analyze the reliability of these scales in a different cohort of patients or in non-hospitalized adults.