Approval for the research was granted by the Institutional Review Board of the Institute of Health, Jimma University (reference number IHRPGI/467/19).

This study was conducted in low-resource primary health care centers, with a specific focus on the maternal and childcare health service units at the Jimma Health Center and the Jimma Higher Two Health Center. Both health centers are situated in Jimma Town, in the Oromia region, Southwestern Ethiopia. Each of them serves up to 40,000 people in its geographical area, accepts referrals from community health posts, and refers patients to the nearest hospital, such as the Shanan Gibe General Hospital and the Jimma University Specialized Hospital. The health centers serve and oversee both inpatient and outpatient cases. The number of personnel in the Jimma Higher Two Health Center is 34 and in the Jimma Health Center is 40, whereas in Ethiopia, the health center’s maternal and child health service unit employs a much smaller number of health professionals, commonly 5-7 nurses and midwives. There were 5 nurses and midwives at the Jimma Health Center and 4 at the Jimma Higher Two Health Center during our investigation. The maternal and childcare health service unit is expected to serve 2000-2500 antenatal care (ANC), pregnant patient care, and postnatal care (PNC) cases annually.

Participants in the CDSS evaluation were health care professionals, such as midwives and nurses, who worked at the maternal and childcare health service unit at the Jimma Health Center and the Jimma Higher Two Health Center. The inclusion and exclusion criteria were as follows:

The CDSS evaluation was conducted in the health care professionals’ spare time because the number of health care professionals at the maternal and childcare health service unit was limited, and they were so preoccupied and busy with their regular daily activities that it was not feasible to incorporate the evaluation into their routine. The health care professionals completed a questionnaire over the course of a half-day (as a summary of the half-day cases rather than as a case-by-case response), with the morning session taking place from 11:00 to1:00 A.M. and the afternoon session taking place from 5.00 to 18:30 P.M.

The initial evaluation was conducted in August 2022 at the Jimma Health Center. The second round of evaluation took place at the Jimma Health Center and the Jimma Higher Two Health Center from December 20, 2022, until January 15, 2023.

Based on our previous experience [13], obtaining the expected sample size in an LRS was difficult due to a shortage of health care professionals in the maternal and childcare health service unit (usually 4-7).

To determine the optimal strategies, we consulted the existing literature in support of our evaluation study design. Based on the findings of Mburu and Oboko’s study [14], we observed that 79 cases were sufficient to assess the use of mobile health (mHealth) interventions in Kenya. Additionally, Mburu and Oboko [14] also reported that 60 subjects were sufficient to detect the small and medium effects of an exogenous latent variable (independent variable) on an endogenous latent variable (dependent variable), according to the findings of Chin and Newsted [15] and Cohen [16], just as using 40 subjects was sufficient for Goodhue et al [17]. The minimum sample size needed to observe an effect with a given power (ie, the probability of observing a statistically significant result at level P if a true effect of a certain magnitude is present) is determined by the effect size. The effect size is associated with the path coefficient between a variable that is assumed to describe a cause and a variable that is assumed to be an effect: values<0.02 indicate no effect, values>0.15 indicate a medium effect, and values>0.35 indicate a large effect [17,18]. Moreover, using 70-80 samples was adequate to model functional brain relationship hypotheses in the study by Sideridis et al [19]. However, Sideridis et al [19] also explicitly noted that sample sizes of 50 participants were associated with a root mean square error of approximation of <0.05, suggesting a satisfactory fit.

The study entailed a proof-of-principle CDSS evaluation using a convenience sample of 7 health professionals. Altogether, we reviewed 73 ANC, pregnant patient, and PNC cases.

A tutorial and a demonstration were provided to the health care professionals at the 2 health centers prior to using the CDSS. The health care professionals used and assessed the CDSS before completing a questionnaire. They used a retrospective chart review, specifically a half-day of pre-recorded patient card sheet data, to evaluate the CDSS. On the basis of pre-recorded cases, the goal was to evaluate how well the CDSS performed in identifying referrals and locally treatable cases that were actually made. The health care professionals then filled out questionnaires to provide their assessments and feedback on the CDSS. Each evaluation questionnaire was completed based on a half-day of ANC, pregnant patient, and PNC cases, as well as the health care professional’s observation of the CDSS reaction to the presented cases. Next, the health care professionals answered a series of 5-point Likert scale items (1=strongly disagree, 2=disagree, 3=neutral, 4=agree, 5=strongly agree) about the CDSS [20]. The measurement instrument consisted of 22 parameters adopted from Ji et al’s [12] evaluation framework. The 22 measurement items were classified into 6 factors: system quality, information quality, service quality, perceived ease of use, user acceptability, and perceived benefits. Furthermore, we automated the questionnaire submission, which was accessed via a mobile phone or a laptop. Electronic questionnaire submission was preferred over paper-based alternatives. However, paper-based questionnaire submissions were used in some cases.

We designed and developed a CDSS to meet the requirements of LRSs. An intelligent clinical wizard, minimum data and data readiness, adaptable features, and low-cost infrastructure are some of the notable requirements and prerequisites of LRSs based on our previous results [7]. Our CDSS incorporates both existing knowledge-based guidelines and data-driven evidence to provide the most relevant information for frontline workers at the time of care delivery [11]. The CDSS provides CPs (or workflows) for point-of-care services. The CP is a critical component of a CDSS for identifying referral and locally treatable cases, which is delivered in the form of a concordance table for multicriteria decision analysis and output [11].

The CDSS has the following major goals:

To get the most out of services, the CDSS provides an automated, interactively adaptable CP (or workflow). To reduce arbitrariness in entry point selection, the CDSS provides a range of choices for initiating the CP, such as using evidence from historical records, dominant factors, or randomly initiating the signs and symptoms based on CGs. Figure 2A presents additional information about entry point processing.

The process is interactive, and our algorithm uses measured symptoms (MSs) and a combination of MSs to process the CPs:

In the end, the frontline worker must make the final decision based on the suggestions made by the algorithm. For this study and demonstration, the CDSS focused on 3 use cases, namely pregnant patient care, ANC, and PNC services. The sample user interface screenshot for each step is shown in Figures 3-7.

Statistical Package for Social Sciences (SPSS; IBM Corporation) version 26.0 [21], Microsoft Excel [22], Python (version 3.7) [23], and SmartPLS (version 26.0) [24] were used to conduct the analysis and modeling.

We followed the procedures and recommendations of Boone and Boone [25] for the CDSS evaluation based on Likert data analysis. Latent variables were computed by summing the following items:

To assess the scale of the CDSS evaluation data set, the validity of the measurement model was checked. Convergent validity was assessed using factor loading and average variance extracted (AVE), with a factor loading threshold of more than 0.70 and an AVE threshold of >0.50 [26,27]. In this study, items with factor loadings of less than 0.70 were candidates for deletion. The internal consistency and reliability of the CDSS evaluation measurement model were assessed using Cronbach α [28] and composite reliability. A recommended value of >.70 for Cronbach α and composite reliability was accepted. We used the heterotrait-monotrait ratio of correlations (HTMT) [26,29] to check the discriminant validity of the measurement model and determined whether the value was less than 0.90 and acceptable. Moreover, perceived benefits are formative second-order construct based on decision changes and process changes. Collinearity was checked to ensure that it did not have a negative impact on the higher-order-construct measurement model, and critical levels of collinearity less than 0.50 were acceptable in this study, as recommended by Hair et al [26].

Following that, item-level and construct-level analyses were performed. On the one hand, an item-level analysis of the CDSS in LRSs between the 2 health centers was conducted. A nonparametric independent-samples statistical test, such as the Mann-Whitney U test [30], was used to see whether the 2 health centers were significantly different at the item level. We used the Mann-Whitney U test because we could not assume normality in either group and the independent data set observation assumptions were fulfilled, which are preconditions for the use of nonparametric data analysis [31]. Furthermore, there were no significant results from the Shapiro-Wilk test [32] on the normality of our evaluation data set. The significance level used for the inferential statistics was P=.05 and a 95% CI level.

On the other hand, we followed the recommendation of Boone and Boone [25] to use Pearson correlation for construct-level (latent variable) correlation analysis. As a result, Pearson correlation [25,33] was used to examine the factors influencing the acceptance of the CDSS in LRSs and the interrelationships between construct factors. In particular, the relationship between system quality and perceived ease of use, information quality and perceived ease of use, user acceptance and perceived benefits, and user acceptance and information and system quality were explicitly explored.

Finally, structural equation modeling (SEM) is a multivariate statistical analysis technique that is used to analyze structural relationships. It is described in the literature as combined factor analysis and regression analysis for discovering relationships between measured variables and latent constructs [34]. There is a debate on how effective it is to discover causation beyond correlation. In papers dealing with applications of the technique, it is commonly used to express a causal hypothesis in a context where there is semantic information available that supports the validity of the hypothesis or at least does not contradict it [12,14,35]. Our study was a pilot study, not a full, cross-sectional analysis, and it intended to promote the use of partial least squares structural equation modeling (PLS-SEM) [26]. PLS-SEM was used to model the acceptance of the CDSS in LRSs, particularly to model the relationship between the CDSS evaluation measured items and construct variables, as well as between multiple construct variables. We noticed that penalized likelihood estimation algorithms based on regularized structural equation modeling (RegSEM) [36,37] and PLS-SEM [26] were the best candidates for our modeling. We preferred PLS-SEM for the following reasons:

Overall, to construct the PLS-SEM model for the CDSS in LRSs, first, composite factor analysis was used to examine the validity of the measurement model, including reliability and validity analysis. The relationships in path models with latent variables were then evaluated using PLS-SEM path analysis and coefficients. Finally, the statistical significance of PLS-SEM results, such as path coefficients, outer weights, Cronbach α, and coefficient of determination (R²) values, was determined using bootstrapping [26]. The bootstrapping settings were percentile bootstrap, 2-tailed test type, and significance level=.05.

The 7 CDSS evaluators were all female (ie, n=4, 57%, from the Jimma Health Center and n=3, 43%, from the Jimma Higher Two Health Center), who worked as health care professionals (eg, midwives and nurses) in the health centers’ maternal and childcare health service units. In total, 73 CDSS evaluation responses were recorded based on 116 ANC, pregnant patient care, and PNC cases (n=4, 5%, during the first evaluation period and n=69, 95%, during the second evaluation period). The response was 73 since the evaluation response was based on a summary of half-day cases rather than a case-by-case response. The average time for evaluating the CDSS and completing the questionnaire was 52.35 minutes, with the smallest and longest durations being 31 and 98 minutes, respectively. The Jimma Health Center accounted for 65.5% (76/116) cases, while the Jimma Higher Two Health Center accounted for 34.5% (40/116) cases. Furthermore, we observed that each health center handled 4-6 (3%-5%) cases per day on average. Overall, the first round of evaluation lasted 2 days and included 18 ANC, pregnant patient care, and PNC cases in the Jimma Health Center, which is above average. In round 2, there were 75 ANC cases, 7 pregnant patient care cases, and 16 PNC cases during our evaluation period. The second round of evaluation took place in both health centers, the Jimma Health Center and the Jimma Higher Two Health Center.

The computer-aided CDSS evaluation’s mean (SD) score ranged from 4.29 (SD 0.485) to 4.52 (SD 0.503). Table 1 provides more extensive details of each item score.

The factor loading of 20 (91%) of 22 items was greater than 0.70. The remaining items, legibility and medical quality, were eliminated since their factor loading value was less than 0.70. All the constructs had Cronbach α values greater than .70, except information quality, for which Cronbach α was .699, which is close to .70. Table 2 provides more information about the measurement model’s construct reliability and validity.

To establish discriminant validity, the HTMT on construct factors was used, and the results showed that all constructs passed the test. Table 3 displays the results of the discriminant validity assessment.

The results of the nonparametric Mann-Whitney U test based on the 5-point Likert item evaluation data set collected from the Jimma Health Center and the Jimma Higher Two Health Center revealed that the 2 health centers did not differ significantly in the CDSS item-level evaluation factors, except stability (U=470.5, P=.022), overall usefulness (U=451.0, P=.012), adherence to standards (U=483, P=.024), confirmation of expectations (U=488.5, P=.04), satisfaction with overall quality (U=400.5, P=.001), and overall satisfaction (U=474.5, P=.023). The findings of the CDSS evaluation using the Mann-Whitney U test are shown in Table 4.

Although the total number of observations in the first and second rounds of the CDSS evaluation were not equal, we found a positive mean agreement score increment in the majority of evaluation parameters at the Jimma Health Center, which was calculated using “agree” and “strongly agree” responses. Adherence to the standards agreement score, however, declined from 1.00 to 0.974. The first and second round CDSS evaluation agreement score observations at the Jimma Health Center are shown in Table 5.

We found a significant correlation (r =0.74) between user acceptance and perceived benefits, with perceived benefits as construct factors based on process changes and decision changes. The coefficient of correlation between perceived ease of use and information quality was r=0.63. User acceptance was also correlated with information quality and system quality, with r=0.68. Figure 8 depicts a more detailed Pearson correlation test result.

The perceived user acceptance coefficient of determination (R²) was 0.703, showing that user acceptance is influenced by system quality, information quality, perceived ease of use, and perceived benefits (Figure 9). More precisely, system quality (β=.321, P<.05) and perceived benefits (β=.486, P<.05) were shown to have a significant influence. However, the perceived ease of use had no positive effect on CDSS acceptance (β=.015, P=.91). Information quality also had no positive effect on CDSS acceptance in this study (β=.149, and P=.25).

Furthermore, we found that the perceived ease of use was influenced by system quality, and information quality, with an R² value of 0.479. The path coefficient of information quality on the perceived ease of use was β=.021(P=.89), and hence, no significant effect was found. The path coefficient of system quality on the perceived ease of use was β=.678 (P<.05), that is, a significant influence, whereas the perceived benefits impacted by decision and process changes had an R² value of 0.983. The path coefficients of decision changes and process changes were β=.374 and β=.749, respectively, and were significant (P<.05). Figure 9 depicts the path weights, P values, and coefficient of determination (R²) for the CDSS evaluation PLS-SEM model developed using the CDSS Jimma Health Center and Jimma Higher Two Health Center evaluation data sets. The results, shown in Figure 9, can be interpreted as perceived ease of use -> perceived user acceptance (β=.015 and P=.91), for example. Overall, we found that the perceived ease of use had no positive effect on CDSS acceptance (β=.015, P=.91) but, rather, on the system quality (β=.321, P<.05) and perceived benefits (β=.486, P<.05) of the CDSS. Further information is presented in Table 6.

This study aimed to evaluate a CDSS for use at the point of care in primary care LRSs. The health care professionals in this study evaluated user acceptance of the CDSS.

The Cronbach α scale of 22 items appeared to be internally consistent, exceeding the minimum value of .70 required for acceptable reliability [26-28,32]. In this study, the 2 health centers did not differ significantly in terms of the CDSS’s perceived ease of use, information quality, and perceived benefits (decision changes and process changes). However, we found a significant difference in system quality, such as stability, and perceived user acceptance, such as overall usefulness, adherence to standards, confirmation of expectations, satisfaction with overall quality, and overall satisfaction. This variation could be attributed to the first round of evaluation, which was based on the Jimma Health Center, or to the fact that more cases were observed in the Jimma Health Center than in the Jimma Higher Two Health Center, but more research and analysis are required. Furthermore, based on the first and second rounds of the CDSS evaluation, we observed a positive agreement score increment at the Jimma Health Center. However, this study was unable to observe a change in the Jimma High Two Health Center, since the first round of evaluation was limited to the Jimma Health Center.

This study highlighted a correlation between construct variables using Pearson correlation. The CDSS’s system quality, information quality, and perceived benefits were vital for its acceptance in the LRS. The perceived benefits were based on decision and process changes. In accordance with our results, previous studies have demonstrated that the acceptance and use of mHealth apps in LRSs are influenced by users’ perceptions that are aligned with health needs and expectations [14]. However, in this study, the perceived ease of use was moderately correlated with CDSS acceptance, whereas Ji et al [12] suggested that the perceived ease of use has a significant and direct impact on the acceptance of a CDSS. Figure 8 depicts further information about the Pearson correlation between the construct variables of the CDSS. Overall, we observed a low positive Pearson correlation between the perceived ease of use and acceptance, as well as between system quality and the perceived ease of use, when we considered the strength of correlation classification as in Mukaka [33]. System quality and acceptance, information quality and acceptance, and information quality and perceived ease of use all showed a moderately positive correlation. There was a high positive correlation between perceived benefits and acceptance, supporting Pande et al’s [40] finding that perceived usefulness is significantly correlated to the intention to use.

This study also used PLS-SEM to evaluate several factors that impact the acceptance of a CDSS in LRSs. The result demonstrated that user acceptance is impacted by system quality, information quality, and perceived benefits, with an R² value of 0.703, as shown in Figure 9. The perceived benefits influenced by decision and process changes had an R² value of 0.983, whereas the R² score for the perceived ease of use as impacted by system and information quality was only 0.479. All retained R² values were greater than 0.10, as suggested by Falk and Miller [41]. The R² values of the perceived user acceptance and perceived benefits were substantial, as also indicated by the CDSS results of Cohen [18], who reported R²>0.26, and Chin [42], who reported R²>0.67. However, according to the criteria of Hair et al [43], R² of perceived benefits is greater than 0.75 and substantial, while R² of the perceived ease of use and user acceptance is greater than 0.50 and moderate. However, Mohamed et al [44] showed that the coefficient of determination must be larger than 0.19, the path coefficient between latent variables must be at least 0.1, and the significance level must be at least .05 in order to validate the model. Our CDSS evaluation model meets all these criteria, except the path coefficient from perceived ease of use to perceived user acceptance, which was 0.015. Hair et al [26], however, stated that path coefficients with standardized values greater than 0.20 are typically significant, while in this study, the path coefficient from perceived ease of use to user acceptance was 0.015, which is less than 0.10 and not significant. More information is depicted in Figure 9, which includes the details of the CDSS assessment PLS-SEM model developed from the CDSS Jimma Health Center and Jimma Higher Two Health Center evaluation data sets, including path weights, P values, and the coefficient of determination (R²).

Overall, as shown in Table 6, the PLS-SEM results suggested that the perceived ease of use has no positive effect on CDSS acceptance (β=.015, P=.91) but, rather, on system quality (β=.321, P=.002) and perceived benefits (β=.486, P<.001) of the CDSS. We also observed that information quality had a positive influence on the perceived ease of use (β=.678, P<.001). However, system quality had no favorable impact on the perceived ease of use (β=.021, P=.89). The detailed conclusions and summary based on PLS-SEM are shown in Table 6.

In this study, we evaluated our own proof-of-principle CDSS in LRSs. The small sample size and low number of cases in our study might limit the generalizability of our findings. As a result, difficulties that were not identified during this investigation may be identified during a longitudinal and case-by-case evaluation.

We designed and developed a CDSS based on LRS requirements, which we evaluated in 2 LRSs in Ethiopia: the Jimma Health Center and the Jimma Higher Two Health Center. Our overall result indicates that user acceptance is impacted by system quality, information quality, perceived ease of use, and perceived benefits, with an R² value of 0.703. Specifically, system quality and perceived benefits have a direct impact on user acceptance of the CDSS in LRSs. In this study, however, we found that the perceived ease of use and information quality had no positive effect on CDSS acceptability. Overall, the proposed acceptance model includes specific factors and variables, which is an important step toward the successful adoption and implementation of a CDSS in LRSs.