Artificial Intelligence (AI) is neither a new nor recent concept. The first time this term was used was back in the 1950s.1 However, it has been in recent years that its large-scale implementation has become possible due to exponential technological growth.2 AI is perceived as a powerful tool with enormous potential to change the way we live; however, AI is no stranger to controversy.3 Among all types of AI, those capable of reaching higher levels of predictive or even creative skills are based on deep learning models.4 These models are nonlinear and rely on a large number of mathematical operations, which complicate their interpretation.5,6 It is precisely this type of AI that generates more concern and distrust in society and is being analyzed from ethical and legal perspectives.7

While the introduction of AI requires a risk-benefit analysis in any scenario, in the health care field, this analysis must be deep and comprehensive. It has been proposed as a solution to many of the problems of current health care systems, contributing to decreasing mortality, shortening the length of stay, or preventing the occurrence of adverse events.8–16

Despite recent efforts to increase the culture of safety in health care systems, adverse events still significantly contribute to morbidity, mortality, and health care spending. The addition of new technologies emerges as a promising strategy in this regard. The critical care setting is an ideal scenario for the implementation of such technologies. Extremely ill patients undergoing numerous procedures and highly complex treatments on a daily basis, which happen to be prone to errors coexist in this context. Additionally, there is pressure for quick decision-making as it is a typical area for time-dependent conditions. The volume of information generated in an ICU can be overwhelming. In fact, it has been suggested that it exceeds the capacity of an expert clinician.17 Thus, the increase in available information in different formats (images, lab test results, genetics, invasive physiological monitoring, etc.) is not always associated with better decisions. Therefore, the addition of AI-derived technologies in critical care areas could help clinicians increase diagnostic or therapeutic capabilities and contribute to improving outcomes by facilitating better integration of information.18 There is growing controversy due to the proliferation of AI publications with poor methodological quality and questionable validity, which limits its implementation. A recent systematic review of more than 400 studies reporting models developed in the critical care setting found that more than 20% were associated with preventing the occurrence of adverse events. However, the authors highlight that the methodological quality of the articles was very poor, most of them being retrospective (96.4%) and highly biased.19

For all these reasons, the aim of this manuscript is to review the potential contribution of AI to patient safety in the critical care setting by summarizing technical aspects and providing examples. Secondly, aspects of safety related to the implementation processes of AI-derived technologies, which will undoubtedly be part of the future of our units, have also been reviewed.

Recently, the use of AI has been proposed in the main domains of adverse events (AE), highlighting prediction, prevention, and early detection of patients at risk of deterioration. In this way, AI - based on the automation of records and the use of ML - offers new strategies to mitigate the occurrence of AE.30 Currently, there are algorithms that use real-time patient data accumulating information to personalize treatment during their admission. These tools, for example, would allow initiating or discontinuing antithrombotic therapy based on the risk of bleeding at a specific time during admission.31 Other algorithms allow summarizing all the information about a patient regarding an event of interest. By condensing it, they facilitate comparison with other patients, as well as the creation of patient cohorts with comparable risk profiles for a specific event of interest (mortality).32 There are multiple clinical settings in which results have been published in this regard, highlighting: 1- prediction and stratification of the risk of readmission, helping in patient flow management and avoiding AE associated with unplanned readmissions.33,34 2- prediction of kidney failure-related AE.35,36 3- prediction of unplanned extubation allowing the implementation of preventive measures and reducing the workload for the health care personnel.37 Although its implementation may focus on mitigating adverse events in ICU patients, it should not be limited to the ICU setting. This technology can be used anywhere in the health care system to predict early clinical deterioration or immediate transfers to the ICU that would allow early treatment initiation and resource organization.38,39 The effectiveness of an algorithm may be limited outside its original environment, as the data it uses reflect the culture and specific practices of each ICU. Consequently, what works in one ICU setting may not be applicable to another, especially if working conditions and staff ratios vary, thus affecting outcomes. The application areas of AI regarding safety are varied, whether inside or outside the ICU sttting (wall-less ICU strategy), and more and more innovative algorithms are becoming available that allow capturing and better adapting the information obtained from patient data to make more precise predictions.

Drug-related incidents remain among the most frequent adverse events.40 Up to 25% of adverse events are considered preventable.41 Furthermore, intensive care units (ICUs), due to their technical complexity and association with time-dependent diseases, are more susceptible to prescription errors.42 The applications of AI in this area are diverse, including risk prediction models for the development of adverse reactions, detection of polypharmacy-related events, development of in silico interaction and allergy models, application of Clinical Decision Support Systems (CDSS), and exploitation of electronic health records for the detection of unintended adverse reactions.43 Depending on when these models are applied, they can help predict risk, reducing their incidence (prevention strategies), or subsequently contribute to early detection, reducing severity and duration (damage mitigation strategies).44

Regarding predictive use cases, various strategies and examples are found in the literature. The most common is the prediction of patients at high risk of adverse drug reactions. The most studied event groups in this regard are renal, cardiovascular, and opioid-related overdoses.44 A second group consists of predicting therapeutic response, which could prevent the use of drugs in non-responsive patients, which is particularly relevant in pharmaceutical groups with a high incidence of adverse events and poor tolerance, such as antineoplastics or some antivirals.45 Predicting the optimal dose is also an area of interest, with suitable drugs for these purposes being anticoagulants or antineoplastics.31,46,47 We should mention that these models integrate information from the patient’s entire medical history, including past health records, current free-text history, or lab test results. Although promising, we should mention that the addition of genetic results into the models has only marginally improved predictive capability.45 From a technical standpoint, the most widely used models are decision trees, natural language processing techniques, and neural networks, although both the variety and variability of these techniques used are extensive.

In the early incident detection block, we distinguish between detecting reactions and detecting medication errors that have already occurred (inappropriate prescription, interactions, and duplications). Undoubtedly, one of the most interesting areas is related to prescription errors. From a safety standpoint, although these occur after an individual action (prescriber, administrator, or consumer), they are considered system failures. Some authors even argue that they should be considered failures of clinical information systems.48 In recent years, some CDSS with ML algorithms have been tested to aid in real-time prescription error detection.49 These models add electronic health record information and subsequently detect prescriptions considered atypical by the model. These atypicalities can be due to infrequent prescriptions (contraceptive for an infant), discordance with the medical history (antidiabetic for a patient without a history of diabetes), or uncommon dosages. These systems can intervene in 2 different way, by generating synchronous (at the time of prescription) or asynchronous alerts (during follow-up when the patient's clinical situation changes). The main types of alerts and examples are described in Table 2.

One of the most developed systems in this regard is the MedAware software, which has been prospectively validated. In a validation on over 78 000 prescriptions, the alert rate was low (0.40%). Of these, 40% were synchronous alerts, with time-dependent alerts being the most frequent (64.80%). Of the generated alerts, 89% were considered appropriate, and 43% resulted in prescription changes. These data were superior to rule-based CDSS, which presented a high alert burden (37.10%) and low clinical significance (5.30% prescription changes).50

As previously mentioned, AI can be useful in various aspects at the ICU setting. However, it also poses challenges, both medical and ethical, as well as technological. Therefore, it is relevant not only to analyze the utility of AI regarding patient safety, but also establish theoretical frameworks for the safe use and implementation processes of AI. After reviewing the possible contributions of this technology to safety, we will now delve into the risks it presents, the possible solutions regarding adverse events it may generate, and the safe implementation of these algorithms.

Prediction algorithms employing supervised machine learning rely on learning from examples. Through them, a system is modeled capable of associating new events with learned data and generate a prediction. It is evident that the data used in learning will be a key point for the model's success. Therefore, data collection is essential, making sure that data accurately represent the target population. The following are among the most cited issues on this regard51:

• •

  Imbalanced populations/cases: this occurs when not all groups are equally represented. If not considered during training, there's a risk of favoring predictions towards one group simply because it includes more cases.

• •

  Non-generalization: this occurs when the population selection for training does not include cases from the entire target population. In this case, if the system goes into production, it will fail to generate predictions for these groups.

• •

  Underrepresentation of a group: as in the previous case, a group is excluded from the training set. This is not a problem in population selection as much as an underrepresentation of this group for social or economic reasons that cannot be solved by broader selection of the training group.

During variable selection, confounding factors should be included, while making sure that variables favoring discrimination of a group have not been included. There are several examples illustrating the importance of this phase. During the COVID-19 pandemic, a lower risk of admissions due to virus-related pneumonias was observed in asthmatic individuals. This phenomenon may be due to risk underestimation in a specific subpopulation, namely, asthmatics developing pneumonia. This underestimation can be attributed to the lack of consideration of relevant factors, such as prior steroid use.52 Additionally, lower rates of heart failure-related admissions have been reported in exclusion risk populations, such as African American and Latino communities, which stresses the need for addressing disparities in risk assessment.53,54 In addition to defining the variables, we will have to select the error function that we want to optimize, i.e., the metric that will eventually evaluate our model. This selection is not trivial and can induce application biases.55 Therefore, it seems evident that it is essential to understand the effect of each evaluation metric on the problem being addressed.56

Various evaluation metrics, their drawbacks, clinical risks, transparency, and use cases are detailed in Table 3. Finally, it is advisable to define what it means for the developed model to be fair, understanding that individuals with similar characteristics are treated similarly.57,58 Several publications propose implementations considering the demographic parity and equality of opportunities; however, their use has not been standardized in the clinical field.59–61

Having an AI generated securely with data from a development environment (DE) does not necessarily imply that it will function securely when implemented as a decision support tool in the routine clinca practice in real-time with data from an implementation environment (IE). It is important to clarify that, although these 2 environments can be different due to spatial dimension (2 different ICUs), they can also differ due to temporal dimension (same ICU, different time periods). Currently, there is no standardized protocol of the steps that should be followed to ensure its success. However, there are consensus guidelines among experts to help us during the process.62 Therefore, we propose a minimum of 4 necessary phases to safely transition an AI from a DE to a IE as a decision support tool (Fig. 2).

Figure 2.

Map of the 4 phases of secure real-time AI implementation. In phase 1, the AI is tested to ensure it adapts to the reality of the data from the center where it is deployed. In phase 2, the data flow is constructed to evaluate its performance in real-time. In phase 3, the results of using AI vs not using it are compared in relation to patient benefit. Finally, in phase 4, the performance of the AI is continuously monitored, and necessary improvements are applied to ensure its evolution for the benefit of all.

(0.32MB).

The addition of artificial intelligence (AI) to the realm of security, while promising, faces key challenges. Predicting adverse events and aiding safe prescription represent significant opportunities. However, the lack of methodological quality in research and the need to address ethical concerns, such as trust and bias, are imperatives. Successful implementation requires not only technical robustness but also careful transition, ensuring understanding and acceptance by health care professionals. Continuous security and adaptability emerge as crucial foundations for effective collaboration between AI and health care, ensuring tangible benefits for patient safety.

During the preparation of this work, the authors used Chat-GPT 3.5 to request synonyms and improve the translation of technical expressions from English to Spanish. After using this service, the authors reviewed and edited the content as necessary, assuming full responsibility for the publication's content.

None declared.

JABM has worked for the artificial intelligence company Savana Médica. The remaining authors declared no conflicts of interest whatsoever.

Jesús Abelardo Barea Mendoza: Conceptualization, drafting, editing, and final manuscript review. Josep Gómez Álvarez: Conceptualization, drafting, editing, and final manuscript review.

Alex Pardo Fernandez: Drafting and final manuscript review.

Marcos Valiente Fernandez: Drafting and final manuscript review.