From a global perspective, many health care systems face comprehensive challenges that affect how care is delivered to society. In this regard, several factors increasingly strain care structures, processes, and the actors involved. For instance, demographic changes and the overall aging of society raise age-related health issues and demands [1,2] and introduce further case complexity; for example, in the form of comorbidity [3]. Simultaneously, a shortage of personnel and medical expertise can be discerned in many—often remote and rural—regions, caused by the low attractiveness of jobs in care due to inappropriate compensation and high workload [4], the attractiveness of urban areas and structures [5], the absence of young graduates willing to establish new or continue existing practices [6], or the trend toward centralized care facilities, inter alia [7]. As a result, larger catchment areas develop for providers who have to cope with deficient and inequitably distributed first-hand access to care [8]. Further, on a societal level, detrimental access to care can marginalize lower socioeconomic groups, as a study from the United States suggests [9], impeding the maintenance of comprehensive and inclusive care. Considering the increasing complexity of medical care on the one hand and the decreasing time and personnel resources on the other hand, the need to actively support clinicians at the point of care is growing.

Representing a promising and widely adopted technology to render processes and decisions more efficient, so-called clinical decision support systems (CDSSs) are software applications capable of catalyzing and informing the process of decision-making of medical professionals [10]. Although applications exist that target the decisional processes of patients, often called decision aids [11] or patient decision support interventions [12], the clinical use of CDSSs remains the primary domain for decision support. Here, the evaluation of performance, adoption, effectiveness, and impact on patient outcomes advances, but still lacks comprehensive approaches [10], including an analysis of relationships among technological characteristics, continual use, and effects on diagnosis and treatment. Nevertheless, the potential of CDSSs to support diagnostic processes leads to their use in other contexts of medicine; for example, primary care [13], and in several different disciplines, from emergency medicine [14] and dermatology [15] to radiology [16]. Aside from diagnostic purposes, CDSSs are used to detect possible inadequate prescriptions of medication [17] or to simulate different treatment strategies and their impact on patient outcomes [18]. Until today, CDSSs have had partial nonadoption for numerous reasons; for example, workflow disturbances or trust deficits, and their adoption is linked to many different factors concerning technology and human-technology interaction [19,20]. In particular, the subjective perception of and attitude toward the CDSS remains a crucial predictor of adoption [21]. This is because the CDSS surpasses the preferably objective description of medical information (eg, in electronic health records) and interprets this information to support clinical interventions [19]. Meanwhile, the comparability of CDSS among different contexts is difficult because of the already-mentioned variation in user groups (patients, physicians, nurses, etc), medical domains (clinical care, primary care, etc), medical disciplines (dermatology, radiology, etc), and purposes (diagnosis, prescription, treatment, etc).

Owing to technological innovations, health care technologies, including CDSSs, are increasingly enabled by artificial intelligence (AI) [22]. The first evaluation of an AI-enabled CDSS promises increased performance and accuracy compared with a conventional CDSS [23]. In addition, clinicians and experts in the field generally expect simplification of organizational processes, such as patient flows, with the advent of AI [24]. Defined as a technology’s capability to work in a way that a human perceives as intelligent [25], AI is used on various occasions with regard to CDSS, such as risk prediction for medical complications [26] and adverse drug effects [27]. However, a rigorous and consistent definition of AI is challenging. Therefore, we followed Helm et al [28] and Schuetz and Venkatesh [29] on their emphasis on the adaptive characteristics of AI, meaning that AI-enabled CDSSs are learning entities that change over time while considering their environmental conditions. Consequently, these systems are not deterministic and may provide different outputs from the same input at different times [30]. Compared with medical professionals, AI-enabled systems can outperform human ratings or predictions; for example, concerning the classification of dermal lesions and proliferation [31]. Regarding the adoption of AI-enabled systems in general, ongoing research reports several concerns indicated by clinicians. Although the fear of being replaced appears to depend on the level of knowledge about the concept of AI that the clinicians possess [32], studies report that clinicians fear being biased by the recommendations of AI, resulting in overconfidence and harmful consequences for patients [33]. In addition, clinicians are concerned that AI might increase the threat of data breaches and the associated risks for patients’ privacy, as well as legal consequences resulting from treatment errors [34]. Nevertheless, current research suggests an ambivalent perception of AI. Considering the aforementioned concerns and potential hindrances for adoption, clinicians assume that AI-enabled systems might save time and improve the continuous monitoring of patients [35]. Furthermore, research has highlighted that only a few clinicians comprehend the variety of applications of AI and its conceptual nature [34,35]. Differences in the perception of AI; for example, regarding the fear of being replaced [36], emphasize the subjectivity of clinicians’ attitude toward AI.

Considering the ambiguity of concerns regarding clinicians’ attitudes toward AI, the mentioned hindrances of CDSS adoption and the similarity between concerns associated with AI and CDSS (eg, biased decision-making, legal consequences, or fear of being replaced), an AI-enabled CDSS might actually increase the relevance of perceptive and subjective factors for adoption and their interplay with technological characteristics. During the process of development and evaluation of AI-enabled CDSS, it became apparent that the potential benefits for clinical performance and treatment quality are maximized by human–AI collaboration, rather than by human-AI competition [31]. However, owing to the interactive and adaptive nature of AI-enabled CDSS, traditional theories and models to explain the use and adoption of these systems forfeit their power to explain and predict a successful collaboration between AI and human beings [29,37]. Specific factors regarding AI-enabled technology and human actors such as dermatologists, radiologists, and other medical professionals are emphasized to influence the relationship among them, including the explainability or understandability of the system [38], its purpose [39], and the resulting trust a human actor perceives in the system [40]. Considering that factors related to the subjective attitude and perception of clinicians, such as trust, already impact the adoption of non-AI CDSS [21,41,42], we argue that the advent of AI-enabled systems increases the importance of specific factors that are not exclusively bound to technological characteristics. Considering the already investigated hindrances impeding the adoption of CDSS by clinicians [43,44], the lack of a sound theoretical basis, or the reliance on traditional theoretical approaches within ongoing research [45], the need for a review of AI-specific factors influencing the collaboration between AI and human actors has increased.

To understand the dyadic relationship between humans and AI, it is necessary to understand key concepts and their interrelations. Although many researchers use the term interaction [46], literature defining what interaction means is seldom. Hornbæk et al [46] showed that there is no common definition and identified 7 concepts of interaction that highlight different perspectives. However, the human–computer interaction framework of Li and Zhang [47] shows that interaction can be generally understood as a process of using a technology for a task in a specific context. In turn, collaboration etymologically stems from the term collaborare which means work with. As the origin reveals, collaboration can be understood as a joint effort in which a common goal is pursued. From our perspective, collaboration is thus a successful interaction with an adaptive AI-enabled system. Under the assumption that both human and AI-enabled systems are not error-free, a human-AI collaboration is effective when errors are prevented. In this context, a key driver of such effective collaboration is that medical professionals perceive the system as trustworthy (ie, a certain level of trust) for the tasks to be done and accept it. Trust is a complex psychological construct that is described as the will to make oneself vulnerable [48]. If a party considers another party to be trustworthy, the relationship is in turn determined by the perception of the other parties’ attributes of ability (the legitimacy of a system’s recommendation for a specific decision), benevolence (the accordance of a human actor’s and the system’s intention and motivation to do good), and integrity (the accordance of a human actor’s and the system’s superordinate values) [40]. Nevertheless, it remains unclear how system design can influence the perception of trustworthiness and what human traits foster the propensity to trust.

The objective of this study is to summarize the factors influencing effective collaboration between medical professionals and AI-enabled CDSSs. Capturing these factors is essential for medical professionals, management, and technology designers to reflect the adoption, implementation, and development of AI-enabled CDSSs [48,49]. Further, we seek to explore what specific outcomes are used to evaluate successful collaboration between humans and AI-enabled CDSSs (performance, effectiveness, impact on patient outcomes, etc) and the theoretical foundations on which they are based. Finally, the comparison between factors that are associated with AI-enabled CDSSs and those associated with CDSSs not enabled by AI appears to be important in evaluating the extent to which the current literature has already reflected the uniqueness of human-AI collaboration.

We conducted a narrative review to summarize the current literature regarding our specific objectives [50]. In the following, we report the search for relevant literature to meet our objective, its selection, and its synthesis to counteract the subjectivity of our results [50]. We selected 3 different meta-databases to search for studies that met our research objective. We defined our search strategy in accordance with the relatively broad scope of our study [51]. To report our results, we followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines for reviews [52]. Through our initial search, we identified 1161 studies by screening titles and abstracts, of which 100 (8.61%) satisfied our inclusion criteria. Through a backward search, we identified another study that was included in our full-text assessment, resulting in 101 articles assessed for eligibility. Finally, 6.9% (7/101) of studies were included in our synthesis of results.

We included the databases PubMed, PsycInfo, and Business Source Complete for our literature review. PubMed indexes >5000 journals in the fields of medicine, health care, and related disciplines. We used PubMed, in particular, to gather information about the clinical effectiveness and implementation of AI-enabled CDSSs. PsycInfo contains >2000 journals from behavioral and social science research. We searched PsycInfo to examine the psychological dimensions of AI-enabled CDSSs and decisional processes. Finally, we scanned results from Business Source Complete, containing >1000 journals in the field of business sciences, to obtain insights regarding our objective from an economic and procedural perspective.

We combined 2 different sections of search terms (AND conditions). The first section represented the technologies associated with the objective of our research (AI OR artificial intelligence OR machine learning OR cognitive computing OR intelligent agent OR decision support OR recommendation agent). The second section reflected the interactional dimension of human-AI collaboration (trust* OR acceptance OR *agreement OR consent OR compliance OR congruency OR collaboration OR resistance). We included articles published in English over the last 10 years. To select relevant literature, 2 authors (MK and SW) independently screened titles and abstracts to exclude articles that did not involve AI-enabled technology (see the definition in the Clinical Decision Support Systems section) and those that were not related to health care or medicine. The inclusion and exclusion criteria were discussed in detail before the screening. In addition, to familiarize themselves with the procedure, an initial sample of 100 entries was screened. A high level of agreement was achieved, and disagreements were resolved through discussion between the 2 authors (MK and SW). In the remaining papers, only a few borderline cases were discussed until consensus was reached, and both the authors (MK and SW) finally came to the same result. In the full-text screening, articles that did not involve AI or AI-enabled systems (n=32), did not consider the interaction between the human actors and AI-enabled systems (n=15), did not distinguish between AI-enabled and non–AI-enabled CDSSs (n=6), did not involve CDSS (n=38) or the perspective of medical professionals (n=1), or appeared to be gray literature or opinion (n=2) were excluded. Detailed documentation of the exclusion process for full-text screening is provided in Multimedia Appendix 1, where all excluded studies and the reasons for exclusion are presented. The selection of relevant literature is represented through a PRISMA flowchart (Figure 1). If articles were eligible, we summarized and reported the specific factors influencing effective collaboration.

On the basis of our study selection, 7 studies were included in our final synthesis. From our perspective, this result stems from the fact that many studies of AI-enabled CDSSs (1) compare solely the diagnostic accuracies of human raters and those of AI-enabled systems and (2) focus on technological characteristics and the development of these systems, but do not discuss their effects on the interaction or collaboration between technology and human actors. Therefore, most (5/7, 71%) of the included studies reflected on the relevance of specific characteristics or factors by contemplating the objective from a meta-perspective (Table 1).

Of the 7 included studies, only 1 (14%) study mentioned the interrelation between an effective human-AI collaboration and primary clinical outcomes. Reviewing an AI-enabled CDSS for skin cancer diagnostics, Felmingham et al [54] mentioned the possible impacts of these systems on a patient’s morbidity and mortality associated with skin cancer in general. Other studies described secondary outcomes, such as a system’s mathematical accuracy [55] or behavioral intentions to use a CDSS [57]. No study investigated the impact of technological characteristics or human factors on primary clinical outcomes.

Of the 7 included studies, only 1 (14%) study mentioned the theoretical foundations on which implications for practice are based explicitly. Jeng and Tzeng [57] derived hypotheses for their empirical investigation from the unified theory of acceptance and use of technology, which is a technology acceptance theory widely adopted to explain the intention to use technology and the subsequent use behavior [61]. An important predecessor in this theoretical model is social influence (ie, “...the degree to which an individual perceives that important others believe he or she should use the new system” [61]). However, based on their results, Jeng and Tzeng [57] discarded their theoretical assumption about social influence affecting clinicians’ intentions to use a CDSS. Felmingham et al [54] discussed the role of cognitive biases in decisional processes involving AI-enabled CDSSs. Nevertheless, Felmingham et al [54] did not explicitly mention the origin of cognitive biases in the prospect theory by Kahneman and Tversky [62].

Our results show that only a few (7/101, 6.9%) studies have already broached the issue of individual factors influencing effective collaboration between a human actor and an AI-enabled CDSS. Although unique considerations with regard to these systems appear; for example, the important role of trust [30,53], scarce empirical evidence exists for the relational structure of essential factors or characteristics. In addition, many studies did not describe the involved system and its characteristics extensively, enabling differentiation between AI and non–AI-enabled systems accurately [42]. Therefore, we argue that a more thorough description of the involved system and its characteristics is highly relevant for future research as it lays the foundation for comparing different systems and their effectiveness. Nevertheless, in the process of reviewing the literature, we were able to differentiate between factors primarily associated with technological structures and functions (technological characteristics), and those primarily associated with human actors’ psychological or perceptual attributes (human factors). Both technological characteristics and human factors influence the nature of the interaction between human actors and AI-enabled CDSSs. Interestingly, some technological characteristics and human factors appear to be antecedents of interaction; for example, the personality of medical professionals [54], whereas others appear to be effects of an interaction [53]. Therefore, as suggested by Felmingham et al [54], it can be assumed that human factors and technological characteristics are mutually dependent and together shape the interaction between human actors and AI-enabled CDSSs. As described in the Background section, the shape of an interaction between human actors and AI and their resulting interactional relationship can be considered a condition for successful collaboration. However, the foundation for evaluating an AI-enabled CDSS differs, in accordance with current research addressing non-AI CDSS [20]. Studies from our results discussed the accuracy or mathematical performance of systems, adoption by medical professionals, sustainability and congruency of interaction, and the effects on patient outcomes to be relevant for evaluation. Although the effectiveness of a collaboration between human actors and AI currently depends on the context and objective of a system [53], the paradigm of medicine clearly dictates the final evaluation of a CDSS by its ability to improve primary and secondary outcomes of patients [22]. As AI-enabled systems are characterized by their adaptive nature [29], processes of individual interaction and collaboration are likely to be iterative and reciprocal and will change and be refined over time. Figure 2 summarizes this process based on our results and can be considered a proposed descriptive framework for human-AI collaboration.

When comparing our results to research concerning medical professionals’ interactions with non-AI CDSS, high correspondence can be noted. Khairat et al [20] mentioned workflow fit (adaptability), computer literacy (technological expertise), trust, a general optimistic attitude of clinicians (personality), and clinical expertise (medical expertise) as important factors for effective adoption. In addition, Khairat et al [20] reported usability and perceived usefulness as determinants. As perceived usefulness needs further concretization within the context of AI-enabled systems [53], usability might generate only minor relevance for AI-enabled CDSSs, as these systems are based on automated processes of use and integrate human-like ways to communicate (eg, natural language processing for voice control) [29]. In contrast, the explainability of a CDSS appears to be a technological characteristic that strongly influences the collaboration between humans and the system, whether it is enabled by AI or not [63]. However, differentiation of explainability and related terms such as understandability, interpretability, and transparency has not yet been completed, and the impact of explainability on other relevant factors, including trust, has not yet been empirically verified [63]. In general, it is not clear how and if technological characteristics and human factors influence other specific aspects of collaboration between human actors and AI-enabled CDSSs. For instance, studies suggest that high clinical expertise influences overall collaborative performance [54] but does not hypothesize possible explanations. Clinical expertise might be associated with a lack of trust in these systems, overconfidence biases, or the fact that these systems are sometimes less accurate than experienced physicians.

Furthermore, other studies involving non-AI CDSSs have emphasized the essential role of trust in the effective interaction between humans and the system. Trust is a multidimensional construct. A lack of trust might result from reservations regarding the mathematical accuracy or appropriateness of a system or the purpose of a system in improving patient outcomes [64]. As our literature review reveals the importance of the technological accuracy of AI-enabled CDSSs, research focusing on trust in human-like technology has shown that ability, benevolence, and integrity are essential prerequisites for sustainable adoption [37,40]. However, only 14% (1/7) of the included studies highlighting trust in its role in successful collaboration defined the actual meaning of trust [30], and none of the included studies paid attention to the prerequisites. Considering the inconsistent definition of trust in technology [48,65], future research might reveal important prerequisites for trust within the interaction between human actors and AI-enabled technologies. In addition, the relationship between trust in AI-enabled CDSSs and improvement in clinical outcomes requires further investigation.

Findings from our literature as well as ongoing research concerning non-AI patient decisional aid suggest that a stronger theoretical foundation for the interaction between human actors and CDSSs is important [66]. Felmingham et al [54] already demonstrated that cognitive biases, originating from the prospect theory, might decisively impact effective collaboration, that is, the tendency to confirm assumptions already made rather than falsify them, known as confirmation bias [67], might distort the relationship between medical professionals and AI-enabled CDSS in the sense that they might not accept a different opinion except their own. In contrast, relying on automated information instead of vigilantly seeking and interpreting information, known as automation bias [68], might actually cause the unreflected acceptance of suggestions made by a CDSS. Therefore, to discuss suitable theoretical foundations, it might be helpful to further explicate and structure the aforementioned nontransparent relations of different constructs, factors, and characteristics influencing decision-making and collaboration. In addition, problems originating from the application of traditional technology-centered theories (such as the technology acceptance model or unified theory of acceptance and use of technology) on AI-enabled decision-making might lead to inappropriate results [29,69]. Theories concerning the trust-based adoption of human-like technology [40] promise to encounter these deficits by emphasizing the interactional components of technology adoption and use.

Our study had some limitations. As some studies derived their conclusions about collaboration between AI-enabled CDSSs and human actors from studies of CDSSs not enabled by AI or assigned results from non-CDSSs to CDSSs, reasoning about interrelations between different technological characteristics and human factors is preliminary and requires further investigation. Although our results fit well with the current findings about the uniqueness and specific nature of human-AI interaction, very few (7/101, 6.9%) studies, of which most were narrative reviews, were included because of our innovative and novel objective as well as the specific context. This may be a result of our relatively narrow search, which could be extended by explicating the related constructs and prerequisites of trust. Explorative empirical studies based on suitable theoretical foundations might yield frameworks and models to structure future research on AI-enabled CDSSs, as our study primarily provides an orientation about relevant individual characteristics and factors. The consideration of environmental influences (eg, organizational policies or culture [30] and patients’ views [70]) on AI-supported decisional processes for medical care is vital for a comprehensible understanding but cannot be provided within the scope of our review.

We extracted the technological characteristics and human factors relevant for effective collaboration between medical professionals and AI-enabled CDSSs. Although most of the findings from previous research on non–AI-enabled CDSSs are in accordance with our results, the weighting of specific factors might change with AI-enabled systems. The adaptive and increasing human-like nature of AI-enabled CDSSs emphasizes the time sensitivity and reciprocity of decisional processes that should ultimately lead to an improvement in care. Cognitive biases may occur at any time during these processes, varying the effectiveness of collaboration. Explainability remains an essential prerequisite for interaction, and the expertise and personalities of medical professionals have come into focus. In addition, trust between humans and the system emerges as a central aspect of decisional support, whereas the interrelations among these facets still need to be investigated. Concepts such as shared decision-making justify the integration of patients’ demands and wishes, an important factor for medical care, and its role in human-AI collaboration is yet underrepresented. Currently, it is unclear how these concepts can be integrated into AI-enhanced decisional processes and to what extent medical decisions with the help of the CDSS are influenced by the subjective meaning and understanding of diagnoses or treatments by patients. In addition, as several studies have measured the effectiveness of collaboration by means of other parameters, primary and secondary patient outcomes should be considered in future research.

As described earlier, modern health care structures are under increasing pressure. Involved medical professionals face immense workloads per capita, and the supply of personnel declines. Because these structures form the initial access points for most citizens in need of care and treatment, approaches that foster more efficient decision-making and treatment processes are becoming imperative to maintain comprehensive care. Thus, an AI-enabled CDSS represents an important and future-oriented measure that enables actors in the health care domain to improve resource allocation, make timelier and less stressful decisions, and cope with shortages in personnel, facilities, and expertise. However, the potential application of CDSSs and pursued benefits calls for investigations that shed light on how AI-enabled processes can be implemented within prevalent health care structures so that the associated risks and challenges, such as the oversimplification of individual patient data or the automated initiation of suboptimal or erroneous treatments, can be mitigated.