The current postgraduate medical training model in most European countries is the so called “time-basedtraining”.1–3 This paradigm assumes that mere exposure to clinical experiences based on temporary rotations through different Departments suffices to acquire the necessary professional competencies. Opportunistic learning and volume of practice rather than learning guided by objectives defines competency. Certification depends on a logbook of cases, and a generic, subjective report about knowledge, technical and non-technical skills acquired by the resident, which is performed after every rotation and yearly by the tutors in charge. A knowledge-based examination is also included in some countries. At present, some prestigious national educational institutions are transitioning from a time-based to a competency-based medical education (CBME) system which is a learner-centered approach that emphasizes achieving specific outcomes.4–8 The CBME model for ICM in Europe is called CoBaTrICE (Competency Based Training in Intensive Care Medicine in Europe).9–14 The implementation of CBME is challenging because requires organizational changes, resources, particularly more dedication of teaching time, as well as the training of tutors and staff members in formative assessment and feedback techniques.15,16 Research in this field is still limited because the model has been applied in a partial and scantly structured manner. In order to evaluate whether the implementation of CoBaTrICE provides higher levels of competency in comparison with the current official time-based program in ICM in Spain, a multicentric cluster-based randomized trial is currently ongoing.17 Before starting the implementation of CoBaTrICE we performed a baseline observational study with the primary objective of determining the level of competency achieved by the residents at the end of the 3rd year of training in ICM. We chose a simulation-based objective structured clinical evaluations (OSCE) to assess the ability to integrate knowledge, judgment, communication, and teamwork into the simulated practice setting.18–23 Secondary objectives were: (1) to identify gaps in performance that could be addressed in future educational interventions, and (2) to investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.

A total of 36 R3 from 13 ICU Departments of 13 Spanish teaching hospitals performed in 176 scenarios. One hospital with two residents eventually declined to participate (Table 2).

Six video-records of scenarios one (1), scenario two (2) and scenario three (3) were lost. The distribution of the participants in the OSCE in each of the four simulation centers was as follows: Barcelona, 9; Madrid, 13; Granada, 6; and Valencia, 8. There were 12 male and 24 female, mean age 29.8±4.4 (range 27–49 years). Half of them had obtained an average grade in their undergraduate studies equal to or higher than B, and half of them had obtained a MIR position lower than number 3000 (range 110–4534). Regarding their previous experience in simulation-based training, 79% had taken a course in advanced cardiopulmonary resuscitation, 25% a course of the management of polytrauma patients, 14% a course of acute crisis resource management, and 72% other simulation-based courses. Fourteen of them were doing their residency in hospitals with more than 1000 beds (five hospitals), fifteen in hospitals with 600–1000 beds (five hospitals), and seven of them in hospitals with less than 600 beds (three hospitals).

Since traditional examinations fail to capture the uncertainty that will be encountered in some clinical scenarios and the assessment of advance skills is difficult in real practice, we created a simulated-based OSCE which was reliable and reproducible across four advanced simulation centers. The total median score achieved by the 36 R3 participants in a total of 176 performances was 71 out of 100 points. The percentage of CEPEs accomplished was 75%. However, only half of the participants achieved the expected level of competency. The experts panel considered that level III (the participant needs supervision to perform the activity in complex situations) should be the appropriate level of competency for that specific stage of training. However, level III was successfully achieved only in the scenario “acute coronary syndrome”. The worst results were obtained in the scenario “management of the septic shock, ARDS and endotracheal intubation”, here, most of the participants applied correctly the 1-hour sepsis bundle and protective mechanical ventilation strategy, however, the approach to the patient was disorganized, and they failed to apply a complete protocol of endotracheal intubation (Additional file 1, Table 1S) in a high-risk patient with severe hypoxemia and shock.27 There was also a room for improvement in the so called non-technical skills such as leadership, setting priorities, and communication with patients and relatives. Although there are not similar studies performed in ICM, our results are not very different from those obtained in studies conducted to assess performance of anesthesia professionals using simulation.19,28 Weinger et al.22 in their study of 268 board-certified anesthesiologists found that CEPEs were commonly omitted. Approximately 30% of encounters were rated as “poor” or as “unsatisfactory” for overall individual technical or behavioral performance. They documented omissions, errors, or delays in actions considered by clinical experts to be critical to successful patient care. As in other studies,19,22,28,29 a wide variability in performance was found. In general, the same resident performed significantly different in some scenarios in comparison to others, there were also differences in performance among the residents of the same hospital. This observed heterogeneity might be due to the characteristics of our traditional experience-based training model where assessments of residents are scarce, indirect and subjective. In this context, it is hard to ensure that residents have the expertise they need to perform entrusted tasks. The CBME model proposes more solid principles such as defining the specific learning outcomes and focusing attention upon the development and demonstration of skills, attitudes and knowledge acquired by residents during the training process. Frequent formative work-based assessments as well as the record of the learning experiences in a portfolio are essential elements to promote learning, self-reflection, progression, and ultimately to guarantee that the predefined competences and skills are effectively acquired.5,30,31

Although the OSCE does provide a standardized and relatively objective method of evaluating a set of clinical skills in medicine, its use does not guarantee accurate decisions about examinees, especially, when referring to the non-technical skills,32 multiple relevant factors such as the number of scenarios, items, scoring, examiners, etc., can influence the results. In addition, although simulation is being widely used to measure technical and non-technical skills21,33–37 there is little documentation of a relationship between simulation performance and performance in the clinical setting.38 It has just been shown that those with more training and experience perform better in the scenarios, suggesting that simulation-based assessments may ultimately prove useful as an indicator that they are ready for unsupervised practice in the real world.19,23,29,39

We designed five 15min clinical scenarios based on the model “crises resource management”,18,40 where rapid decision-making based on incomplete data is needed. Each participant performed as primary intensivist and worked in a team with trained confederate clinicians and nurses to provide an environment as much realistic as possible. Although the simulated clinical environment was not identical to the participants usual one, where they would probably perform better, they “strongly agreed” in the post-assessment survey the scenarios were realistic, the duration was appropriate, and the competences assessed were relevant to their clinical practice, what further supports construct-related validity. Like in other studies, previous experience in simulation did not influence the level of competency achieved for the residents.19,22 Finally, the definitions of the levels of performance were stablished by a consensus of a panel of experts, thus, they are debatable.

Despite these limitations, we still think this study provides a pathway to identify gaps in performance in common problem areas.

The expected level of competency after a year in the ICU was achieved only in half of the performances. Gaps were observed in compliance with evidence-based protocols and also in non-technical skills. Reliance on the traditional experience-based training model alone might be insufficient for ensuring quality and safety in patient care. A more evidence-based educational approach is needed.

Multiple center simulation-based assessment showed feasibility, validity and reliability as an evaluation method of competency of the residents in the scenarios. It could be replicated for formative assessments, and even for nationwide comparisons and performance benchmarking. Additional research is needed to determine how simulation-based assessments predict performance in clinical settings.

ACO and RGR had full access to all the data in the study and took responsibility for the integrity of the data and the accuracy of the data analysis. ACO and RGR were responsible for the study design, data analysis, and interpretation, and the writing of the manuscript. MCFD and MDSB were responsible for the statistical analysis. ACO, MJB, DPC, VGT, MV, CV, IM, NM, ES, MJP were responsible for selection the CoBaTrICE competences to be assessed, design the scenarios, define the items of the checklist for each scenario, organization and implementation of the OSCE in the four simulation centers, assessment and rating the performances. ACO was responsible for participants recruitment. All authors read and approved the final manuscript.

The study has received funding from:

• 1.

  ESICM Trials Group Award 2018. Trials Group Awards. European Society of Intensive Care Medicine.

• 2.

  Conselleria de Educación, Investigación, Cultura y Deporte. Generalitat Valenciana. Código proyecto AICO/2018/126.

The authors declare that they have no competing interests.