This paper is in the following e-collection/theme issue:

Usability Evaluation Case Studies (182) Registered Report (516) Mobile Health (mhealth) (2748) Usability of Apps and User Perceptions of mHealth (892) Cardiology (126) Mobile Apps for Cardiology (127) Pediatrics (121) Human Factors and Usability Case Studies (739) Innovations in Cardiopulmonary Resuscitation (31)

Published on in Vol 9, No 1 (2022): Jan-Mar

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/35399, first published .
Usability Testing and Technology Acceptance of an mHealth App at the Point of Care During Simulated Pediatric In- and Out-of-Hospital Cardiopulmonary Resuscitations: Study Nested Within 2 Multicenter Randomized Controlled Trials

Usability Testing and Technology Acceptance of an mHealth App at the Point of Care During Simulated Pediatric In- and Out-of-Hospital Cardiopulmonary Resuscitations: Study Nested Within 2 Multicenter Randomized Controlled Trials

Usability Testing and Technology Acceptance of an mHealth App at the Point of Care During Simulated Pediatric In- and Out-of-Hospital Cardiopulmonary Resuscitations: Study Nested Within 2 Multicenter Randomized Controlled Trials

Authors of this article:

Johan N Siebert1, 2 Author Orcid Image ;   Laëtitia Gosetto3 Author Orcid Image ;   Manon Sauvage3 Author Orcid Image ;   Laurie Bloudeau4 Author Orcid Image ;   Laurent Suppan2, 5 Author Orcid Image ;   Frédérique Rodieux2, 6 Author Orcid Image ;   Kevin Haddad1 Author Orcid Image ;   Florence Hugon1 Author Orcid Image ;   Alain Gervaix1, 2 Author Orcid Image ;   Christian Lovis2, 7 Author Orcid Image ;   Christophe Combescure2, 8 Author Orcid Image ;   Sergio Manzano1, 2 Author Orcid Image ;   Frederic Ehrler3 Author Orcid Image ;   PedAMINES Trial Group9 ;   PedAMINES Prehospital Group9
Article Authors Cited by (1) Tweetations (4) Metrics

    Original Paper

    1Department of Pediatric Emergency Medicine, Geneva Children’s Hospital, Geneva University Hospitals, Geneva, Switzerland

    2Faculty of Medicine, University of Geneva, Geneva, Switzerland

    3Diagnostic Department, Geneva University Hospitals, Geneva, Switzerland

    4A.C.E. Geneva Ambulances SA, Geneva, Switzerland

    5Department of Emergency Medicine, Geneva University Hospitals, Geneva, Switzerland

    6Division of Clinical Pharmacology and Toxicology, Department of Anesthesiology, Clinical Pharmacology, Intensive Care and Emergency Medicine, Geneva University Hospitals, Geneva, Switzerland

    7Division of Medical Information Sciences, Department of Radiology and Medical Informatics, Geneva University Hospitals, Geneva, Switzerland

    8Division of Clinical Epidemiology, Department of Health and Community Medicine, Geneva University Hospitals, Geneva, Switzerland

    9See Acknowledgments

    *these authors contributed equally

    Corresponding Author:

    Johan N Siebert, MD

    Department of Pediatric Emergency Medicine

    Geneva Children’s Hospital

    Geneva University Hospitals

    Avenue de la Roseraie, 47

    Geneva, 1205

    Switzerland

    Phone: 41 795534072

    Email: Johan.Siebert@hcuge.ch


    Background: Mobile apps are increasingly being used in various domains of medicine. Few are evidence-based, and their benefits can only be achieved if end users intend to adopt and use them. To date, only a small fraction of mobile apps have published data on their field usability and end user acceptance results, especially in emergency medicine.

    Objective: This study aims to determine the usability and acceptance of an evidence-based mobile app while safely preparing emergency drugs at the point of care during pediatric in- and out-of-hospital cardiopulmonary resuscitations by frontline caregivers.

    Methods: In 2 multicenter randomized controlled parent trials conducted at 6 pediatric emergency departments from March 1 to December 31, 2017, and 14 emergency medical services from September 3, 2019, to January 21, 2020, the usability and technology acceptance of the PedAMINES (Pediatric Accurate Medication in Emergency Situations) app were evaluated among skilled pediatric emergency nurses and advanced paramedics when preparing continuous infusions of vasoactive drugs and direct intravenous emergency drugs at pediatric dosages during standardized, simulation-based, pediatric in- and out-of-hospital cardiac arrest scenarios, respectively. Usability was measured using the 10-item System Usability Scale. A 26-item technology acceptance self-administered survey (5-point Likert-type scales), adapted from the Unified Theory of Acceptance and Use of Technology model, was used to measure app acceptance and intention to use.

    Results: All 100% (128/128) of nurses (crossover trial) and 49.3% (74/150) of paramedics (parallel trial) were assigned to the mobile app. Mean total scores on the System Usability Scale were excellent and reached 89.5 (SD 8.8; 95% CI 88.0-91.1) for nurses and 89.7 (SD 8.7; 95% CI 87.7-91.7) for paramedics. Acceptance of the technology was very good and rated on average >4.5/5 for 5 of the 8 independent constructs evaluated. Only the image construct scored between 3.2 and 3.5 by both participant populations.

    Conclusions: The results provide evidence that dedicated mobile apps can be easy to use and highly accepted at the point of care during in- and out-of-hospital cardiopulmonary resuscitations by frontline emergency caregivers. These findings can contribute to the implementation and valorization of studies aimed at evaluating the usability and acceptance of mobile apps in the field by caregivers, even in critical situations.

    Trial Registration: ClinicalTrials.gov NCT03021122; https://clinicaltrials.gov/ct2/show/NCT03021122. ClinicalTrials.gov NCT03921346; https://clinicaltrials.gov/ct2/show/NCT03921346

    International Registered Report Identifier (IRRID): RR2-10.1186/s13063-019-3726-4

    JMIR Hum Factors 2022;9(1):e35399

    doi:10.2196/35399

    Keywords

    cardiopulmonary resuscitation; drugs; emergency medical services; medication errors; mobile health; mobile apps; out-of-hospital cardiac arrest; paramedics; pediatrics; System Usability Scale; Unified Theory of Acceptance and Use of Technology; smartphone; mobile phone 


    Background

    Over the past few decades, health information technologies (HITs) and communication technologies have been widely adopted in health care environments to improve care provision, efficiency, quality, and patient safety while achieving cost savings [1]. Supported by the rapid spread of mobile devices and their innovative features (eg, connectivity, on-board computing capabilities, small size, and operating systems allowing mobile app development), mobile health (mHealth) has undergone considerable development to address health issues by providing medical and communication services within easy reach of end users [2-4]. As of the first quarter of 2021, approximately 5.7 million apps were available on leading web-based app stores [5]. Among them, >325,000 are mHealth apps [6]. The vast majority (65%) are wellness apps designed to be used primarily by the general public. Approximately 15% are patient-centered apps and focus on self-management of specific conditions, and the remaining 20% are medical apps intended for health care providers [7]. Unfortunately, most do not adhere to relevant medical evidence and lack expert involvement in their development or validation process through high-quality studies to support their adoption in clinical practice [2,8-13]. Even when apps are evidence-based, this does not guarantee that they will be used consistently over time. As with other HIT, their benefits can only be achieved if end users intend to adopt them [10,14].

    Understanding users’ HIT adoption behavior is a long-standing topic in the literature and could lead to improvements in the acceptability and use of mHealth apps. Several models have been proposed to predict and understand users’ acceptance and use of HIT. Two of the most commonly used models are the Technology Acceptance Model (TAM) [15,16], or its extended versions (TAM Task–Technology Fit [TTF] [17], TAM2 [18], and TAM3 [19]), and the Unified Theory of Acceptance and Use of Technology (UTAUT; version 1 [20] or 2 [21]). However, a systematic review identified that mobile apps were the least frequently studied HIT application areas using the TAM, with only 15 studies published between 1989 (ie, the year when the TAM was introduced) and 2017 [22]. Although the UTAUT has a better explanatory power for technology acceptance [23] and has been extensively used in studies related to the adoption of HIT, user acceptance and use of medical apps from the perspective of caregivers have been little investigated by this model [24]. In addition, even if usability has been identified as a key component of good practice in the development of digital apps [25], the number of medical apps that publish their usability evaluation results remains scarce [26]. To this end, the validated System Usability Scale (SUS) has been identified as the most frequently used questionnaire [27-29]; however, it concerns mostly mobile apps designed to support patients in health self-management and not caregivers [26]. As a result, little is known about health professionals’ willingness to implement and use medical apps in clinical care [26,30,31].

    Previous Work

    In previous randomized trials, we reported fewer medication errors and a shorter time to drug preparation and delivery during out-of-hospital cardiac arrest (OHCA) and in-hospital post–cardiac arrest scenarios when using the PedAMINES (Pediatric Accurate Medication in Emergency Situations) app than conventional preparation methods [32,33]. This evidence-based app was designed as a step-by-step guide for the preparation and delivery of intravenous drugs to address the unmet need for reducing pediatric medication errors [34]. Recent findings showed that this app was also able to reduce acute perceived stress while preparing emergency drugs in a prehospital setting during pediatric OHCA in a simulated model [35]. However, its usability and technology acceptance by frontline caregivers remains to be determined.

    Aim

    This study aims to investigate the usability of the PedAMINES app by both advanced paramedics with drug preparation autonomy and nurses at the point of care to gain insight into their perceptions of its adoption as an approach to facilitate emergency drug preparation in pediatric life-threatening situations. In addition, we measure the technology acceptance of the app for its intended purpose and user satisfaction with its use. We hypothesize that this approach would help estimate the likelihood of adoption of the app for implementation among its future target users.


    Study Design

    This is a nested, overlapping study within the context of 2 prospective, multicenter, randomized controlled trials registered at ClinicalTrials.gov (NCT03021122 and NCT03921346). The parent trials had the broader and primary aim of assessing rates of medication dosing errors during simulation-based pediatric OHCA and in-hospital post–cardiac arrest scenarios using a high-fidelity manikin [32,33]. For that purpose, participants were randomly assigned (1:1 ratio) to prepare the drugs either with the support of the app (intervention group) or by conventional methods (control group). The trial protocols containing details of the scenarios have been previously published [36,37]. Both trials were performed in accordance with appropriate guidelines [38,39] and followed the CONSORT (Consolidated Standards of Reporting Trials) reporting guidelines [40].

    Technology Acceptance Terminology and Definition

    The terms technology acceptability, acceptance, and adoption are often used confusingly or interchangeably in the mHealth literature. For the purposes of this paper, the term technology acceptance is used and should be understood as referring to initial acceptance of use, as recently defined by Nadal et al [41], to explicitly distinguish between the different temporal stages of technological acceptance. It refers to users’ first interactions with the app at the preadoption stage before sustained acceptance of its use in the postadoption stage.

    Setting and Intervention

    The first trial [32] was conducted at 14 emergency medical services (EMSs). An out-of-hospital child’s bedroom environment was simulated at each EMS center to resemble, as much as possible, a standard environment where paramedics would have to intervene. Participants were tested on sequential preparations of four direct intravenous emergency drugs of varying degrees of preparation difficulty (epinephrine, midazolam, 10% dextrose, and sodium bicarbonate) during simulated pediatric OHCA. The second trial [33] was conducted as a crossover study at 6 pediatric emergency departments. Participants were tested on the preparation of continuous infusions (dopamine and norepinephrine) during simulated pediatric immediate post–cardiac arrest scenarios. Procedures were standardized across all sites to follow the same chronological progression and range of difficulty to ensure that each participant was exposed to exactly the same case in their respective setting, with similar challenges in app use and decision-making. The scripted and uniform delivery of the scenarios throughout the studies was strictly preserved to minimize confounders. Importantly, we did not organize pretests to minimize preparation bias so as not to influence the usability and acceptability of the app based on previous experiences. In both trials, the app was interfaced on an Apple iPad with the latest version of iOS; however, the app works identically on the Android OS (Google Inc). In both trials, all participants who had used the app were surveyed with self-administered questionnaires (refer to the following sections) immediately after completing the scenarios, with the necessary precautions taken to ensure that participants could not communicate with each other. During completion of the questionnaires, no interaction occurred between the participants and investigators other than those related to detailing an item upon the request of a participant.

    Participants

    All registered paramedics and pediatric nurses working at 14 EMSs and 6 academic and community Swiss pediatric emergency departments, respectively, were eligible for inclusion in the study. Participants were of both sexes, of all ages, of different levels of pediatric experience, and from different regions of Switzerland, a pluralistic country with several official languages (French, German, and Italian), to diversify participant characteristics and limit selection bias. Apart from language translation, no other transcultural adaptations were required or made to the app. The inclusion criteria were to have followed a standardized 5-minute introductory course on the use of the mobile device app and written informed consent. This introductory course was not intended at this stage to test the usability of the app but to explain its use for the upcoming intervention. Only participants who had used the app (ie, all those enrolled in the in-hospital study [crossover trial] and half of those enrolled in the out-of-hospital study [parallel group trial]) were eligible for this study. All participants were assumed to have equivalent experience and competence with intravenous drug preparation and dose calculation, as this is part of their regular practice and training background. Given that this study was nested within both randomized trials, the number of participants queried in each trial stemmed from the power calculations set for the primary outcome, with a 2-sided risk α of .05 and a power of 90%. Blinding to the purpose of each trial during recruitment was maintained to minimize the preparation bias. Participants were unblinded after randomization at the beginning of the scenario. Although the intervention could not be masked, all investigators remained unaware of the outcomes until all data were unlocked for analysis at the end of the trial.

    Measurement Instruments

    On the day of participation after randomized allocation and before scenarios started, each participant was required to complete a survey collecting data regarding their demographic characteristics and health care training. Five-point Likert-type scales, ranging from 1=strongly disagree to 5=strongly agree, were used to assess (1) their experience in the use of smartphones and tablets, (2) satisfaction with current supports at their disposal to prepare emergency intravenous drugs, (3) perceived mastery of the preparation of these drugs, (4) propensity to use technological tools in emergency situations, and (5) attitude toward the introduction of technological tools to facilitate the preparation of intravenous drugs during an emergency. Each participant was then exposed to the simulated out-of-hospital or in-hospital scenarios.

    After the scenarios were completed, the perceived usability of the app was measured using the reliable SUS designed by Brooke [27]. According to the International Organization for Standardization, the SUS assesses effectiveness (ie, the ability of users to use the product), efficiency (ie, the effort to use the product), and satisfaction (ie, how the users felt when using the product). It comprises a 10-item questionnaire with 5 response options for each item based on their level of agreement, ranging from 1 (strongly disagree) to 5 (strongly agree). According to the scoring system by Brooke, for odd-numbered (1, 3, 5, 7, and 9) statements (the positively worded items), the score contribution is equal to the scale position minus 1 (eg, strongly agree: 5 – 1=4). For even-numbered (2, 4, 6, 8, and 10) statements (the negatively worded items), the score contribution is equal to 5 minus the scale position (eg, strongly agree: 5 – 5=0). Each score contribution falls within the range of 0 to 4. The participants’ scores for each item are then added up together and multiplied by 2.5 to convert the original scores of 0 to 40 to 0 to 100. Although the scores range from 0 to 100, these are not percentages of usability. The higher the score, the better the usability (ie, 0=very poor perceived usability and 100=excellent perceived usability). To obtain an SUS score of 100, the respondent must answer 5 to all odd questions and 0 to all even questions. The original SUS items are presented in Multimedia Appendix 1 [27]. When translating the SUS questionnaire from its original version to French, German, and Italian [42,43], we replaced the general term system with the specific term PedAMINES.

    Acceptability and usability testing of the app was also assessed in both trials using a tailored 26-item technology acceptance self-administered survey. Scales and queries with high levels of internal consistency (ie, Cronbach α >.70) were derived and slightly adapted from prior research to fit the trial’s context (Multimedia Appendix 2) [44]. This integrative model gathers the 2 most commonly used models in the literature that inform technology acceptance [45], namely the TAM [15] and UTAUT [20,21], as well as additional dimensions of technology acceptance models [19,20,46-50], with the following eight core constructs: (1) perceived usefulness (4 items), (2) perceived ease of use (4 items), (3) TTF (4 items), (4) performance expectancy (3 items), (5) impact on image (2 items), (6) personal innovativeness (3 items), (7) acceptance (3 items), and (8) behavioral intention to use the technology (3 items). The items measured the constructs by asking participants to agree or disagree with statements using 5-point Likert-type scales, ranging from strongly disagree (1) to strongly agree (5). An item of the image construct (ie, Query 34, Image Expectancy 1; People in my organization who use the tool have a high profile), as well as the constructs job security and facilitating conditions, were dropped as they were irrelevant in the context under study. The final survey is depicted in the Results section. According to the original UTAUT model, gender, age, experience, and voluntariness of use are identified as moderators that affect technology beliefs and use [20]. Among these, we expected age to influence the attitude toward the use of the app, knowing that young adults have been more exposed to new technologies during their education and daily practice than older adults. Therefore, we selected age as a factor of interest to correlate with the technology acceptance self-administered survey items. Older participants were expected to have a greater reluctance in introducing the app into their daily practice.

    Finally, a question using a 10-point Likert scale was administered to participants to measure their perceived satisfaction with the use of either the app or conventional preparation methods to prepare the drugs during the resuscitation scenarios (on a scale of 1 to 10, how satisfied were you with your preparation experience?). Data collection was conducted on site using Microsoft Excel spreadsheet (version 2011; Microsoft Corporation) and REDCap (Research Electronic Data Capture; Vanderbilt University) database. The investigators double-checked that the questionnaires were fully and accurately completed on site. Only study investigators had access to the data.

    Statistical Analysis

    Age and work experience were assessed using the Spearman correlation coefficient. The mean global SUS score was reported using SD and 2-sided 95% CIs. Items on the SUS questionnaire were described using frequencies. For the technology acceptance self-administered survey questionnaire, scores on the 8 technology acceptance dimensions were described using the means, SDs, and frequencies of participants with a score of ≥4. The items were described in a similar manner. The association between dimension scores and participant ages was examined using the Spearman correlation coefficient. Paramedics and nurses were analyzed independently, without any comparison made. Statistical tests on the correlation coefficients were 2-tailed, with a significance level of 5%. Data analysis was conducted using R for Windows (version 4.0.2; R Core Team).

    Ethics Approval

    Both trials received a declaration of no objection by the Geneva Cantonal Ethics Committee and Swiss Ethics as their purpose was to examine the effect of the intervention on health care providers. Both trials were registered at ClinicalTrials.gov (NCT03021122 and NCT03921346). The trials were conducted in accordance with the principles of the Declaration of Helsinki [51], Good Clinical Practice guidelines [52], the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online TeleHealth; Multimedia Appendix 3) [39], and the Reporting Guidelines for Health Care Simulation Research [38].


    Overview

    A total of 202 participants using the app (74, 36.6%, paramedics and 128, 63.4%, nurses) completed the scenarios and questionnaires without dropouts. Table 1 summarizes their demographic and health care characteristics.

    Table 1. Baseline characteristics (N=202).
    CharacteristicsParamedics (n=74)Nurses (n=128)
    Age (years), mean (SD)35.7 (7.3)37.2 (9.7)
    Age (years), n (%)

    		<3017 (23)36 (28.1)

    		30-3935 (47.3)41 (32)

    		40-4918 (24.3)35 (27.3)

    		≥504 (5.4)16 (12.5)
    Gender, n (%)

    		Female25 (33.8)121 (94.5)

    		Male49 (66.2)7 (5.5)
    Work experience in years since certification, mean (SD)7.8 (6)13.6 (9.1)
    Work experience in years since certification, n (%)

    		<526 (35.1)23 (18)

    		5-926 (35.1)24 (18.8)

    		10-1915 (20.3)47 (36.7)

    		≥207 (9.5)34 (26.6)
    Smartphone user, n (%)

    		No0 (0)5 (3.9)

    		Yes74 (100)123 (96.1)
    Comfortable using a smartphone, n (%)

    		Strongly disagree1 (1.4)14 (12.1)

    		Disagree3 (4.1)32 (27.6)

    		Neither disagree nor agree12 (16.2)55 (47.4)

    		Agree38 (51.4)15 (12.9)

    		Strongly agree20 (27)0 (0)

    		Missing data0 (0)12 (9.4)
    Last preparation of emergency drugs (months)a, n (%)

    		Never9 (12.2)57 (44.5)

    		<615 (20.3)10 (7.8)

    		6-1217 (23)13 (10.2)

    		12-2415 (20.3)12 (9.4)

    		>2018 (24.3)36 (28.1)
    Satisfaction with the media currently available to prepare emergency drugs, n (%)

    		Strongly disagree9 (12.2)9 (7.1)

    		Disagree18 (24.3)23 (18.3)

    		Neither disagree nor agree25 (33.8)63 (50)

    		Agree21 (28.4)26 (20.6)

    		Strongly agree1 (1.4)5 (4)

    		Missing data0 (0)0 (0)
    Mastering the preparation of emergency drugs, n (%)

    		Strongly disagree9 (12.2)35 (27.3)

    		Disagree15 (20.3)37 (28.9)

    		Neither disagree nor agree30 (40.5)35 (27.3)

    		Agree19 (25.7)17 (13.3)

    		Strongly agree1 (1.4)4 (3.1)

    		Missing data0 (0)0 (0)
    In favor of introducing technological tools to assist in emergency drug preparation, n (%)

    		Strongly disagree0 (0)1 (0.8)

    		Disagree0 (0)0 (0)

    		Neither disagree nor agree6 (8.1)4 (3.1)

    		Agree24 (32.4)45 (35.2)

    		Strongly agree44 (59.5)78 (60.9)

    		Missing data0 (0)0 (0)

    aFor the nurses, emergency drugs meant vasoactive drugs in continuous infusion.

    Although there was a wide age distribution, most respondents were aged between 30 and 49 years, with the nursing group having slightly more older participants. A strong correlation (r=0.72; P<.001) between paramedics’ age and work experience, expressed in years, was found, as well as a very strong correlation (r=0.95; P<.001) for nurses.

    Usability Testing

    As shown in Figure 1, the mean total SUS scores for the app were 89.7 (SD 8.7; 95% CI 87.7-91.7) for paramedics and 89.5 (SD 8.8; 95% CI 88.0-91.1) for nurses’ quotation, which qualifies the tool as between excellent and the best imaginable, according to Bangor et al [53]. All scores were at least >60, spanning from 62.5 (2/202, 1% of people) to 100 (26/202, 12.9% of people; Figure 2). Figure 3 provides a visual overview of the distribution of the item responses on the SUS. SUS total score was not significantly associated with participants’ age (paramedics: r=−0.05 and P=.66; nurses: r=−0.01 and P=.91), which is likely related to highly skewed scores toward high usability with little variability among respondents (Multimedia Appendix 4). The other demographic moderators detailed in Table 1 did not have any significant effect on participants’ acceptance of the app.

    ‎
    Figure 1. Overall System Usability Scale (SUS) scores to assess the usability of the PedAMINES (Pediatric Accurate Medication in Emergency Situations) app. The SUS score is located on a normalized scale ranging from a minimum score of 0 to a maximum of 100 [27]. Adjective ratings provide an interpretation of the SUS score [53]. The SUS also provides letter grades, similar to those used in the traditional school grading system [54]. The acceptability ranges indicate whether the tool is acceptable or not. Red dots represent the mean SUS score in paramedics and blue dots in nurses. Capped blue and red lines represent the 5th and 95th percentiles. Crosses represent medians (paramedics: 92.5, 5th-95th percentiles: 74.125-100; nurses: 90, 5th-95th percentiles: 72.5-100).
    View this figure
    ‎
    Figure 2. Distribution of counts of System Usability Scale (SUS) total scores. Red dots denote paramedics; blue dots denote nurses.
    View this figure
    ‎
    Figure 3. Percent distribution of item responses by (A) paramedics (n=74) and (B) nurses (n=128) on the (inversed) System Usability Scale (SUS) items. The SUS comprises 10 items (numbered as SUS1 to SUS10).
    View this figure

    Technology Acceptance

    Table 2 shows the overall perceptions of paramedics and nurses regarding the mobile app according to the adapted technology acceptance self-administered survey constructs. The paramedics and nurses largely agreed (mostly with scores ≥4) that the app (1) could enhance their performance during emergency drug preparation (perceived usefulness), (2) was easy to use (perceived ease of use), (3) met the requirements necessary to effortlessly address the complexity of the drug preparation task when handled by a technology tool (TTF), (4) could help them achieve performance gains during this procedure (performance expectancy), (5) elicited their intention to use it (attitude toward using a technology), (6) is acceptable for emergency drug preparation (acceptance), and (7) would be intentionally used for this purpose over a long period (intention to use). Intention to use the app was the highest-rated construct, with the item Assuming I had access to PedAMINES, I intend to use it receiving the highest agreement among participants (Table 2). The paramedics and nurses agreed to a lesser extent that the app would enhance the adopter’s social image or status in their organization. Participants had the least agreement on the item Having PedAMINES will be a status symbol in my organization from the image construct. As with the SUS, the technology acceptance self-administered survey scores were skewed toward high scores of acceptance of the app and intention to use it. Only the constructs image, attitude toward technology use, and acceptance showed a somewhat greater spread across the score range. A weak but significantly negative correlation between nurses’ age and attitude toward app use (r=−0.27; P=.003) was identified. Although the TAM used here or the original UTAUT from which it was derived can be influenced by four key moderating variables (ie, age, experience, gender, and voluntariness of use), the effect of the latter two on the acceptance constructs was not analyzed in this study, given the lack of significant variation in these moderators across individuals in the same setting [55]. Experience was highly correlated with age and, therefore, was not analyzed. No significant correlation between age and other items of the technology acceptance self-administered survey constructs was found, and no correlation was observed among paramedics. Multimedia Appendix 5 presents the results of the technology acceptance self-administered survey items by score range.

    Satisfaction

    The app obtained a mean overall satisfaction score of 9.1 (SD 0.9) out of 10 among paramedics and 9.4 (SD 1.0) among nurses.

    Table 2. Results of the technology acceptance survey by items.
    Constructs and itemsDefinitiona and item wordingParamedics (n=74)Nurses (n=128)


    		Values, meanb (SD)Score ≥4, n (%)Values, mean (SD)Score ≥4, n (%)
    PUcThe degree to which an individual perceives that using the system leads to enhanced personal performance [15,18]4.69 (0.44)66 (89.1)4.79 (0.37)122 (95.3)

    		PU1Using PedAMINESd helps me to prepare emergency drugs more quickly.4.69 (0.62)68 (91.9)4.88 (0.32)128 (100)

    		PU2Using PedAMINES helps me to prepare emergency drugs better.4.73 (0.58)71 (95.9)4.74 (0.52)123 (96.1)

    		PU3Using PedAMINES makes it easier for me to prepare emergency drugs.4.69 (0.52)72 (97.3)4.81 (0.51)126 (98.4)

    		PU4Using PedAMINES enhances my effectiveness in drug preparation.4.66 (0.50)73 (99)4.74 (0.61)121 (94.5)
    PEOUeThe degree to which an individual perceives that using the system will be free from physical or mental efforts [15,18,56]4.61 (0.35)72 (97.3)4.76 (0.28)126 (98.4)

    		PEOU1It is easy to get PedAMINES to do what l want it to do.4.41 (0.94)63 (85.1)4.78 (0.47)125 (97.7)

    		PEOU2Overall, I find PedAMINES is easy to use.4.74 (0.47)73 (98.6)4.86 (0.35)128 (100)

    		PEOU3It is easy for me to become skillful in using PedAMINES.4.89 (0.31)74 (100)4.88 (0.33)128 (100)

    		PEOU4I often become confused with PedAMINES’ features when I used it.4.41 (0.55)72 (97.3)4.53 (0.59)124 (96.9)
    TTFf,gThe degree to which an individual perceives that using the system fits the requirements of a particular task [17,44,48]4.49 (0.54)63 (85.1)4.64 (0.43)121 (95.3)

    		TTF1PedAMINES has the functionalities l need to accomplish my tasks.4.55 (0.62)71 (95.9)4.69 (0.52)126 (98.4)

    		TTF2PedAMINES’ functionalities give me exactly what I need for my work.4.47 (0.67)67 (91)4.61 (0.54)125 (97.7)

    		TTF3PedAMINES is very well suited to my work.4.70 (0.54)71 (95.9)4.77 (0.44)127 (99.2)

    		TTF4Using PedAMINES is compatible with most aspects of my work.4.22 (0.90)65 (87.8)4.50 (0.68)116 (91.3)
    PEhThe degree to which an individual perceives that using the system will help the user attain gains in job performance [20,44]4.55 (0.59)61 (82.4)4.58 (0.57)115 (89.8)

    		PE1Using PedAMINES, I get better chances to improve my professional position.4.28 (0.96)59 (79.7)4.21 (1.0)101 (78.9)

    		PE2Using PedAMINES will help me improve or continue to help to improve emergency drugs preparation.4.61 (0.74)67 (90.5)4.77 (0.55)125 (97.7)

    		PE3Using PedAMINES will increase the quality of my drug preparation.4.74 (0.57)71 (95.9)4.77 (0.52)124 (96.9)
    IIi,jThe extent to which an individual perceives that using the system enhances one’s image or status in ones’ social system [20,50]3.46 (0.96)30 (40.5)3.24 (0.98)41 (32.5)k

    		II1People in my practice setting who use PedAMINES will have more prestige than those who do not.3.84 (1.06)47 (63.5)3.53 (1.13)71 (55.9)l

    		II2Using PedAMINES will be a status symbol in my practice setting.3.08 (1.16)22 (29.7)2.94 (1.09)36 (28.6)k
    PIm,nThe extent to which an individual has an innate propensity (willingness) toward trying any new technology [50,57]4.14 (0.58)48 (64.9)3.97 (0.77)73 (57.5)l

    		PI1If I heard about a new technology, I would look for ways to experiment with it.4.55 (0.58)71 (95.9)4.39 (0.73)116 (91.3)l

    		PI2Among my peers, I am usually the first to try out new technologies.3.59 (0.95)42 (56.8)3.42 (1.03)62 (48.8)l

    		PI3I like to experiment with new technologies.4.28 (0.67)65 (87.8)4.09 (0.85)102 (80.3)l
    Ao,pThe extent to which individuals accept to use a new technology [56]4.32 (0.61)56 (75.7)4.30 (0.63)100 (79.4)k

    		A1In my opinion, it would be desirable to use PedAMINES in addition to conventional preparation methods for emergency drugs.4.53 (0.74)69 (93.2)4.40 (0.90)111 (88.1)k

    		A2It would be good to use PedAMINES more than the conventional methods for the preparation of emergency drugs.4.39 (0.70)67 (90.5)4.45 (0.78)114 (89.8)l

    		A3I think it would be highly desirable to use only PedAMINES instead of conventional methods for the preparation of emergency drugs.4.04 (1.07)54 (73)4.06 (1.14)94 (74)l
    BIq,rIndividuals’ subjective intention toward using a technology over a longer period [21]4.81 (0.34)72 (97.3)4.82 (0.39)125 (98.4)l

    		BI1Assuming I had access to PedAMINES, I intend to use it.4.68 (0.60)69 (93.2)4.81 (0.41)126 (99.2)l

    		BI2I predict I would use PedAMINES in the next 6 months.4.91 (0.29)74 (100)4.85 (0.42)126 (99.2)l

    		BI3I expect my use of PedAMINES to continue in the future.4.84 (0.37)74 (100)4.80 (0.44)125 (98.4)l

    aPresented with the source references from which the items were derived and adapted for the context of this study.

    bEach item was measured on a 5-point Likert-type scale (1=strongly disagree and 5=strongly agree); the higher the score, the more agreement with the statement.

    cPU: perceived usefulness.

    dPedAMINES: Pediatric Accurate Medication in Emergency Situations.

    ePEOU: perceived ease of use.

    fTTF: task–technology fit.

    gMissing data from nurses, n=1.

    hPE: performance expectancy.

    iII: image.

    jMissing data under nurses, n=2.

    kN=126.

    lN=127.

    mPI: personal innovativeness.

    nMissing data under nurses, n=1.

    oA: acceptance.

    pMissing data under nurses, n=2.

    qBI: behavioral intention to use.

    rMissing data under nurses, n=1.


    Principal Findings

    The main finding of our study was that the usability assessment of the PedAMINES app scored high on all items of the SUS questionnaire, with >86% of paramedics and >93% of nurses rating the overall usability of the app as excellent (ie, scoring >80 on the SUS). This suggests that the app is highly usable by users targeted for its purpose and appears to be an accepted supportive tool. This usability observed regardless of years of experience, age, or gender suggests a worthwhile benefit of its use by novice emergency care providers and those with limited exposure to children who are critically ill and few opportunities to prepare emergency medications in pediatric doses, particularly in general hospitals handling pediatric emergencies and in the prehospital setting. Leveling providers’ compounding skills could indeed prove to be an advantage in high-stakes clinical events with infrequent occurrences, such as pediatric cardiopulmonary resuscitation. To our knowledge, this is the first study to report usability testing and the intent of paramedics and nurses to use a mobile app as a supportive digital tool for emergency drug preparation in pediatric life-threatening situations. The fact that participants who were previously naive about the PedAMINES app were able to use it correctly to significantly reduce medication error rates [32,33] after a single 5-minute prescenario training and broadly agree on its usability validated the user-centered design [58-60], the Fitts laws [61], and progressive disclosure [62] principles that underpinned its iterative development process [63]. This approach has proven beneficial in allowing end users to influence the development process and increase the final usability of web-based HIT [64,65]. To provide some perspective, the mean total SUS scores >89 in this study were higher than those reported in a comparative study of the top 10 nonmedical mobile apps evaluated by >3500 users [66]. The SUS has proven to be a highly robust and versatile tool for collecting users’ subjective evaluations of a product’s usability [67]. In high-stakes, critical situations where time is of the essence, the usability of dedicated apps must be high as there is no room at that moment to become familiar with the app [68]. This study suggests that endowing paramedics and nurses with usability-proven mobile apps might contribute to improving the safety of the drug administration process in pediatric emergency care.

    To limit and mitigate the likelihood of medication errors, several assistive eHealth technologies have been developed over the past decades to target and support individual medication steps [34]. However, before an eHealth tool such as a mobile app can be adopted and implemented into clinical practice to support the delivery of health care, usability testing and the likelihood that it will be accepted as an aid by its future users are prerequisites for success [69,70]. Usability (ie, the extent to which a system, product, or service can be used by end users to achieve specific goals with effectiveness, efficiency, and satisfaction in a given context of use [71]) is recognized as a key quality factor in determining the success and adoption of an app [26]. Usability testing of an app with end users early in the process may also uncover potential issues related to poor app development and design, which could otherwise ultimately lead to endangering patient safety [72,73]. Although previous research has shown that mHealth apps can help improve the quality and safety of care [74], functional characteristics have often been privileged to the detriment of the needs and characteristics of end users [23]. Consequently, users may be reluctant to adopt them or only use them for a short span of time after their introduction and then abandon them [75,76]. To fully anticipate their acceptance and long-term adoption rate, it is essential to look beyond the technology itself and its usability by also considering end users’ beliefs, perceptions, and intentions regarding its use [24].

    To date, only a small fraction of mHealth apps have published their usability evaluation results, with most apps developed in the commercial sector that have rarely been published in the scientific literature [26]. In the field of cardiopulmonary resuscitation, a recent study identified 34 available mobile apps on Google Play and Apple App stores [68]. However, many of these apps are marginally medically correct, have not been validated in evidence-based studies, and have limited usability [77]. Of the few available medical mobile apps that offer weight-based drug dosing, such as Handtevy Mobile [78], SafeDose Mobile (eBroselow) [79], Infinite Dose PRO [80], Pedi QuikCalc [81], PEDeDose [82], and EZDrips Peds [83], none have shown proven results of their efficacy, especially in terms of usability.

    Although studies have been conducted in recent years to assess user attitudes toward new HIT, there is a lack of research on the perception and acceptance of mHealth technologies by health care providers [84]. In this study, we evaluated 8 technology acceptance constructs to capture the different aspects that could drive the intention to use the PedAMINES app from the perspective of emergency care professionals. We report that paramedics and nurses had an overall acceptant attitude toward the app and agreed with its use as a means of supporting the drug preparation process during emergency care. Among the identified constructs that drive technology acceptance, participants strongly agreed with the usefulness, ease of use, TTF, performance, and behavioral intention to use the app, with mean total scores >4.5/5. In addition, attitudes toward use and acceptance were positive. As these constructs have been shown to be among the most important factors driving individuals to adopt mHealth apps [69,75,85,86], the results of our study provide support for the future adoption of our app by nurses and paramedics in emergency care. However, long-term adoption studies in clinical practice need to be conducted to confirm this. One of the strengths of the study is that the evaluation of the app did not focus on the behavioral intent of its adoption under traditional laboratory-based conditions but took into account the usual context of its use by evaluating its usability and acceptability in simulated, in situ conditions very close to the reality in which the app should ultimately be deployed [73].

    Our study also found that nurses’ attitudes toward app use were slightly negatively influenced by age, whereas usability ratings were not. Consistent with previous research, this may be as some form of resistance may characterize the attitude toward the use of new technologies among nurses with longer experience and their reliance on their own prior learning without the use of technology to assist them [87-89]. Paramedics were represented in younger age categories than nurses, which may explain the lack of a significant age influence on their attitude toward the app. However, previous studies on the effect of age on attitudes toward novelty have yielded controversial results, and caution should be exercised about this interpretation related to the influence of age, and this aspect requires further study [87]. Of note, our study also found that images appeared to be the least relevant technology acceptance construct. Similar to a previous study [90], this could be explained by the nature of the intervention, as the impact of an app on the image of its users among their peers can presumably only be properly assessed through long-term use in routine practice and not through a single use, as was the case in this study. Future research should determine whether the acceptability of the app, as assessed through this simulation-based study, will translate into its long-term use in real-life situations, as the success of an app likely depends primarily on sustained use over time rather than on its first use [91,92]. Meanwhile, this study generated useful knowledge to guide future developments of other mHealth interventions using in situ high-fidelity simulation and the SUS and technology acceptance self-administered survey questionnaires as a basis for exploring the usability and acceptability of mobile apps at the point of care in emergency medicine.

    Limitations

    This study has some limitations. First, the app was evaluated in simulation-based trials, and its usability and acceptance in real-life situations might be different. However, the simulated scenarios allowed for consistent assessment of the app among participants in a standardized manner, which would have been difficult to achieve in actual cardiopulmonary resuscitations. In situ clinical simulation, which involves observing representative users performing tasks in their representative environments, has proven to be a valuable way of addressing specific factors that influence technology usability and adoption and ensuring that the results obtained can be generalized to the real world [73]. Second, the survey used to assess the acceptance of the app was not the original UTAUT model but an adapted version with additional valid and reliable constructs. Adding constructs to the original framework is a common practice [93]. As suggested by Venkatesh et al [21], the addition of relevant constructs to the original UTAUT model, where each construct can be interpreted independently of the others, may contribute to extending its applicability to other contexts in an attempt to better understand factors influencing adoption and behavioral intention to use a technology. For example, the TTF construct was used here to evaluate the users’ perception of the complexity of the task to be handled by being supported by the app. Third, this study was limited to the evaluation of the app in the context of drug preparations at pediatric doses. It does not provide information on its usability and acceptance if it were to be used to support the preparation of drugs for adults. Fourth, no qualitative usability testing was conducted in the field as part of this study so as to not detract from the scenario and primary outcome. More extensive qualitative laboratory and field testing of the app will be the focus of future research. Finally, the SUS questionnaire measured users’ perceived usability of the app rather than its actual usability. Users were not informed of their performance after the scenarios and only filled out the questionnaire based on their perception of their app use. Future studies must consider assessing objective PedAMINES usability metrics, such as task completion rate and efficiency on time on task.

    Conclusions

    This study evaluated the usability and technology acceptance of a mobile app to assist in the preparation of emergency drugs at the point of care by skilled nurses and advanced paramedics during simulated pediatric cardiopulmonary resuscitations. We found excellent usability and high technology acceptance. This provides information not only about the initial adoption of the app by these caregivers but also, more broadly, about the likelihood of successful adoption of mHealth apps in emergency care that have previously gone through such a usability and technology acceptance evaluation process. Our findings contribute to the exploration of factors influencing the usability and acceptance of mHealth apps by emergency caregivers, particularly when devolved to pediatrics, and lay the groundwork for future clinical practice and long-term research.

    Acknowledgments

    The authors are grateful to Rosemary Sudan (Freelance Technical Editor) for providing editorial assistance. The authors are also grateful to Mr Lorenz Vogt and Mr Alexandre Glasner for their assistance in translation at some emergency medical service (EMS) centers. They were compensated for their work.

    This study was funded by the Swiss National Science Foundation (fund SNSF_32003B_169348 and SNSF_32003B-182374). The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    The collaborating members of the PedAMINES (Pediatric Accurate Medication in Emergency Situations) prehospital group are as follows: Philippe Cottet, MD (ACE Geneva Ambulances); Marec Saillant, EMS paramedic (Geneva Team Ambulances); Renaud Grandjean, EMS paramedic (SK Ambulances); Annick Leuenberger, EMS paramedic (Secours Ambulances Genève); Pascal Donnet, EMS paramedic, and Philippe Hauck, EMS paramedic (Service de Sauvetage et de Lutte contre les Incendies Aéroportuaires); Sébastien Pappalardo, EMS paramedic (Service d’Incendie et de Secours); Philippe Nidegger, EMS paramedic (Ambulance Riviera); David Neel, flight EMS paramedic (Air Zermatt SA); Stephan Steinhauser, MD (Höhere Fachschule für Rettungsberufe); Michel Ceschi, EMS paramedic, and Bruno Belli, EMS paramedic (Servizio Ambulanza Locarnese e Valli); Sébastien Ottet, EMS paramedic, and Wenceslao Garcia, MD (Ambulances du Sud Fribourgeois); Yoan Mollier (EMS paramedic), Yves Vollenweider (EMS paramedic), and Pierre Voumard (EMS paramedic; Service Communal de la Sécurité); Karine Corbat, EMS paramedic, and Philippe Robadey, EMS paramedic (Service de Protection et Sauvetage Lausanne); and Joël Bauer, EMS paramedic, and Cyril Berger, EMS paramedic (Centre de Secours et d’Urgences Morges-Aubonne). These contributors coordinated the trial in their respective EMS centers. None of them received compensation for their role in the study.

    The collaborating members of the PedAMINES trial group are as follows: Mario Gehri, MD, and Corine Yersin, registered nurse (RN; HEL, Lausanne University Children’s Hospital); Daniel Garcia, MD, and Franziska Hermann Marina, RN (Inselspital, Bern University Children’s Hospital); Cosette Pharisa, MD, and Monika Spannaus, RN (Fribourg Children’s Hospital); Laurence Racine (MD), Bernard Laubscher (MD), and Carole Vah (RN; Neuchâtel Children’s Hospital); Juan Llor, MD, and Aude Juzan, RN (Sion Children’s Hospital). These contributors coordinated the trial in their respective pediatric emergency department centers. None of them received compensation for their role in the study.

    Authors' Contributions

    JNS was responsible for the concept and design of the study, coordinated the overall conduct of the trials at each center, was responsible for the literature search, interpreted the data, prepared the figures and tables, and drafted the manuscript. LG and MS contributed to the drafting of the manuscript and critical review of its content with the support of CC, LS, FR, FE, and SM. LB, SM, and JNS were responsible for the data acquisition. CC performed the data analyses. KH and FH provided technical and material support. FE was responsible for the development of the project software with the support of SM and JNS. CL and AG oversaw the development of the project software. SM was the main trial coordinator. SM procured funding for the prehospital trial, whereas AG procured funding for the in-hospital trial. JNS was responsible for the overall conduct of this study and oversaw the writing of this manuscript. All authors have contributed to, seen, and approved the final submitted version of the manuscript; had full access to all the data, including statistical reports and tables in the study; and take responsibility for the integrity of the data and the accuracy of the data analysis. The corresponding author (JNS) confirms that he had full access to the participants’ data and endorsed the final responsibility for the submission. He further affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any deviations from the study plan have been explained.

    Conflicts of Interest

    Geneva University Hospitals are owners of the PedAMINES (Pediatric Accurate Medication in Emergency Situations) app. The app is currently commercially available on Google Play Store and Apple App Store for research and educational purposes. JNS, CL, AG, FE, and SM declare individual intellectual property rights on this app and, as employees of Geneva University Hospitals, indirect institutional rewarding through its commercialization (ie, without personal enrichment). The authors declare no other relationships or activities that could appear to have influenced the submitted work. All authors have completed the International Committee of Medical Journal Editors uniform disclosure form and declare no support from commercial entities for the submitted work and no financial relationships with any commercial entities that might have an interest in the submitted work in the previous 3 years.

    ‎
    Multimedia Appendix 1

    The System Usability Scale questionnaire.

    DOCX File , 17 KB
    ‎
    Multimedia Appendix 2

    The technology acceptance survey for the evaluation of health professionals’ acceptance of a mobile app (PedAMINES [Pediatric Accurate Medication in Emergency Situations]) for pediatric drug preparation.

    DOCX File , 24 KB
    ‎
    Multimedia Appendix 3

    CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online TeleHealth) checklist (version 1.6.1).

    PDF File (Adobe PDF File), 415 KB
    ‎
    Multimedia Appendix 4

    Results of the System Usability Scale items for all participants by occupation type (paramedics or nurses).

    DOCX File , 18 KB
    ‎
    Multimedia Appendix 5

    Results of the technology acceptance survey items by occupation type (paramedics or nurses) and score ranges.

    DOCX File , 16 KB
    1. Kruse CS, Beane A. Health information technology continues to show positive effect on medical outcomes: systematic review. J Med Internet Res 2018 Feb 05;20(2):e41 [FREE Full text] [CrossRef] [Medline]
    2. Magrabi F, Habli I, Sujan M, Wong D, Thimbleby H, Baker M, et al. Why is it so difficult to govern mobile apps in healthcare? BMJ Health Care Inform 2019 Nov 18;26(1):e100006 [FREE Full text] [CrossRef] [Medline]
    3. Ryu S. Book review: mHealth: new horizons for health through mobile technologies: based on the findings of the second global survey on ehealth (global observatory for ehealth series, volume 3). Healthc Inform Res 2012;18(3):231. [CrossRef]
    4. Ventola CL. Mobile devices and apps for health care professionals: uses and benefits. P T 2014 May;39(5):356-364 [FREE Full text] [Medline]
    5. Number of apps available in leading app stores as of 1st quarter 2021. Statista.   URL: https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/ [accessed 2022-02-19]
    6. Baxter C, Carroll J, Keogh B, Vandelanotte C. Assessment of mobile health apps using built-in smartphone sensors for diagnosis and treatment: systematic survey of apps listed in international curated health app libraries. JMIR Mhealth Uhealth 2020 Feb 03;8(2):e16741 [FREE Full text] [CrossRef] [Medline]
    7. Jones M, Morris J, Deruyter F. Mobile healthcare and people with disabilities: current state and future needs. Int J Environ Res Public Health 2018 Mar 14;15(3):515 [FREE Full text] [CrossRef] [Medline]
    8. Subhi Y, Bube SH, Rolskov Bojsen S, Skou Thomsen AS, Konge L. Expert involvement and adherence to medical evidence in medical mobile phone apps: a systematic review. JMIR Mhealth Uhealth 2015 Jul 27;3(3):e79 [FREE Full text] [CrossRef] [Medline]
    9. Buijink AW, Visser BJ, Marshall L. Medical apps for smartphones: lack of evidence undermines quality and safety. Evid Based Med 2013 Jun;18(3):90-92. [CrossRef] [Medline]
    10. Gordon WJ, Landman A, Zhang H, Bates DW. Beyond validation: getting health apps into clinical practice. NPJ Digit Med 2020;3:14 [FREE Full text] [CrossRef] [Medline]
    11. Labrique A, Vasudevan L, Chang LW, Mehl G. H_pe for mHealth: more "y" or "o" on the horizon? Int J Med Inform 2013 May;82(5):467-469 [FREE Full text] [CrossRef] [Medline]
    12. Boulos MN, Brewer AC, Karimkhani C, Buller DB, Dellavalle RP. Mobile medical and health apps: state of the art, concerns, regulatory control and certification. Online J Public Health Inform 2014;5(3):229 [FREE Full text] [CrossRef] [Medline]
    13. Van Velthoven MH, Smith J, Wells G, Brindley D. Digital health app development standards: a systematic review protocol. BMJ Open 2018 Aug 17;8(8):e022969 [FREE Full text] [CrossRef] [Medline]
    14. Strudwick G. Predicting nurses' use of healthcare technology using the technology acceptance model: an integrative review. Comput Inform Nurs 2015 May;33(5):189-98; quiz E1. [CrossRef] [Medline]
    15. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 1989 Sep;13(3):319. [CrossRef]
    16. King WR, He J. A meta-analysis of the technology acceptance model. Inf Manag 2006 Sep;43(6):740-755. [CrossRef]
    17. Dishaw MT, Strong DM. Extending the technology acceptance model with task–technology fit constructs. Inf Manag 1999 Jul;36(1):9-21 [FREE Full text] [CrossRef]
    18. Venkatesh V, Davis FD. A theoretical extension of the technology acceptance model: four longitudinal field studies. Manag Sci 2000 Feb;46(2):186-204. [CrossRef]
    19. Venkatesh V, Bala H. Technology acceptance model 3 and a research agenda on interventions. Decision Sciences 2008 May;39(2):273-315. [CrossRef]
    20. Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. MIS Quarterly 2003;27(3):425. [CrossRef]
    21. Venkatesh V, Thong JY, Xu X. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS Quarterly 2012;36(1):157. [CrossRef]
    22. Rahimi B, Nadri H, Lotfnezhad Afshar H, Timpka T. A systematic review of the technology acceptance model in health informatics. Appl Clin Inform 2018 Jul;9(3):604-634 [FREE Full text] [CrossRef] [Medline]
    23. Kim S, Lee K, Hwang H, Yoo S. Analysis of the factors influencing healthcare professionals' adoption of mobile electronic medical record (EMR) using the unified theory of acceptance and use of technology (UTAUT) in a tertiary hospital. BMC Med Inform Decis Mak 2016 Jan 30;16:12 [FREE Full text] [CrossRef] [Medline]
    24. Jacob C, Sanchez-Vazquez A, Ivory C. Understanding clinicians' adoption of mobile health tools: a qualitative review of the most used frameworks. JMIR Mhealth Uhealth 2020 Jul 06;8(7):e18072 [FREE Full text] [CrossRef] [Medline]
    25. Zapata BC, Fernández-Alemán JL, Idri A, Toval A. Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst 2015 Feb;39(2):1. [CrossRef] [Medline]
    26. Maramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform 2019 Jun;126:95-104. [CrossRef] [Medline]
    27. Brooke J. SUS: a 'quick and dirty' usability scale. In: Usability Evaluation In Industry. London: Taylor and Francis; Jun 11, 1996.
    28. Brooke J. SUS: a retrospective. J Usab Stud 2013 Feb 1;8(2):29-40.
    29. Sousa V, Lopez KD. Towards usable e-health. Appl Clin Inform 2017 Dec 21;08(02):470-490. [CrossRef]
    30. Mayer MA, Rodríguez Blanco O, Torrejon A. Use of health apps by nurses for professional purposes: web-based survey study. JMIR Mhealth Uhealth 2019 Nov 1;7(11):e15195. [CrossRef]
    31. Kong T, Scott MM, Li Y, Wichelman C. Physician attitudes towards-and adoption of-mobile health. Digit Health 2020;6:2055207620907187 [FREE Full text] [CrossRef] [Medline]
    32. Siebert JN, Bloudeau L, Combescure C, Haddad K, Hugon F, Suppan L, Pediatric Accurate Medication in Emergency Situations (PedAMINES) Prehospital Group. Effect of a mobile app on prehospital medication errors during simulated pediatric resuscitation: a randomized clinical trial. JAMA Netw Open 2021 Aug 02;4(8):e2123007 [FREE Full text] [CrossRef] [Medline]
    33. Siebert JN, Ehrler F, Combescure C, Lovis C, Haddad K, Hugon F, PedAMINES Trial Group. A mobile device application to reduce medication errors and time to drug delivery during simulated paediatric cardiopulmonary resuscitation: a multicentre, randomised, controlled, crossover trial. Lancet Child Adolesc Health 2019 May;3(5):303-311. [CrossRef] [Medline]
    34. Ehrler F, Siebert JN. PedAMINES: a disruptive mHealth app to tackle paediatric medication errors. Swiss Med Wkly 2020 Aug 24;150:w20335 [FREE Full text] [CrossRef] [Medline]
    35. Lacour M, Bloudeau L, Combescure C, Haddad K, Hugon F, Suppan L, PedAMINES Prehospital Group. Impact of a mobile app on paramedics' perceived and physiologic stress response during simulated prehospital pediatric cardiopulmonary resuscitation: study nested within a multicenter randomized controlled trial. JMIR Mhealth Uhealth 2021 Oct 07;9(10):e31748 [FREE Full text] [CrossRef] [Medline]
    36. Siebert JN, Bloudeau L, Ehrler F, Combescure C, Haddad K, Hugon F, et al. A mobile device app to reduce prehospital medication errors and time to drug preparation and delivery by emergency medical services during simulated pediatric cardiopulmonary resuscitation: study protocol of a multicenter, prospective, randomized controlled trial. Trials 2019 Nov 20;20(1):634 [FREE Full text] [CrossRef] [Medline]
    37. Siebert JN, Ehrler F, Lovis C, Combescure C, Haddad K, Gervaix A, et al. A mobile device app to reduce medication errors and time to drug delivery during pediatric cardiopulmonary resuscitation: study protocol of a multicenter randomized controlled crossover trial. JMIR Res Protoc 2017 Aug 22;6(8):e167 [FREE Full text] [CrossRef] [Medline]
    38. Cheng A, Kessler D, Mackinnon R, Chang TP, Nadkarni VM, Hunt EA, International Network for Simulation-based Pediatric Innovation‚ Research‚Education (INSPIRE) Reporting Guidelines Investigators. Reporting guidelines for health care simulation research: extensions to the CONSORT and STROBE statements. Simul Healthc 2016 Aug;11(4):238-248. [CrossRef] [Medline]
    39. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions. J Med Internet Res 2011 Dec 31;13(4):e126 [FREE Full text] [CrossRef] [Medline]
    40. Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010 Mar 23;340:c332 [FREE Full text] [CrossRef] [Medline]
    41. Nadal C, Sas C, Doherty G. Technology acceptance in mobile health: scoping review of definitions, models, and measurement. J Med Internet Res 2020 Jul 06;22(7):e17256 [FREE Full text] [CrossRef] [Medline]
    42. Gao M, Kortum P, Oswald FL. Multi-language toolkit for the system usability scale. Int J Human Comput Interaction 2020 Aug 19;36(20):1883-1901. [CrossRef]
    43. Gronier G, Baudet A. Psychometric evaluation of the F-SUS: creation and validation of the French version of the system usability scale. Int J Human Comput Interaction 2021 Mar 17;37(16):1571-1582. [CrossRef]
    44. Acceptance factors and use of information technologies: an empirical study on the use of "Rational Suite" software by the employees of a large IT services company. Quebec University (UQAM).   URL: https://archipel.uqam.ca/3960/1/M11905.pdf [accessed 2022-02-20]
    45. Heinsch M, Wyllie J, Carlson J, Wells H, Tickner C, Kay-Lambkin F. Theories informing eHealth implementation: systematic review and typology classification. J Med Internet Res 2021 May 31;23(5):e18500 [FREE Full text] [CrossRef] [Medline]
    46. Goodhue DL, Thompson RL. Task-technology fit and individual performance. MIS Quarterly 1995 Jun;19(2):213. [CrossRef]
    47. Strong DM, Dishaw M, Bandy D. Extending task technology fit with computer self-efficacy. In: Eighth Americas Conference on Information Systems. 2002 Presented at: Eighth Americas Conference on Information Systems; Aug 8-11, 2002; Dallas, United States. [CrossRef]
    48. Klopping IM, McKinney E. Extending the technology acceptance model and the task-technology fit model to consumer e-commerce. Inform Technol Learn Perform J 2004;22(1):35-48.
    49. Moore GC, Benbasat I. Development of an instrument to measure the perceptions of adopting an information technology innovation. Inf Syst Res 1991 Sep;2(3):192-222. [CrossRef]
    50. Yi M, Jackson J, Park J, Probst J. Understanding information technology acceptance by individual professionals: toward an integrative view. Inf Manag 2006 Apr;43(3):350-363 [FREE Full text] [CrossRef]
    51. World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA 2013 Nov 27;310(20):2191-2194. [CrossRef] [Medline]
    52. ICH Harmonised Tripartite Guideline. Statistical principles for clinical trials. International Conference on Harmonisation E9 Expert Working Group. Stat Med 1999 Aug 15;18(15):1905-1942. [Medline]
    53. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009 May 01;4(3):114-123. [CrossRef]
    54. Sauro J. A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices. Scotts Valley, California, US: CreateSpace Independent Publishing; 2011.
    55. Dwivedi YK, Rana NP, Jeyaraj A, Clement M, Williams MD. Re-examining the unified theory of acceptance and use of technology (UTAUT): towards a revised theoretical model. Inf Syst Front 2017 Jun 8;21(3):719-734. [CrossRef]
    56. Klopping I, McKinney E. Extending the technology acceptance model and the task-technology fit model to consumer e-commerce. Inform Technol Learn Perform J 2004;22(1):35-48.
    57. Agarwal R, Prasad J. A conceptual and operational definition of personal innovativeness in the domain of information technology. Inf Syst Res 1998 Jun;9(2):204-215. [CrossRef]
    58. Durrani S, Durrani QS. Applying cognitive psychology to user interfaces. In: Proceedings of the First International Conference on Intelligent Human Computer Interaction. 2009 Presented at: Proceedings of the First International Conference on Intelligent Human Computer Interaction; Jan 20–23, 2009; Allahabad, India   URL: https://doi.org/10.1007/978-81-8489-203-1_14 [CrossRef]
    59. Oviatt S. Human-centered design meets cognitive load theory: designing interfaces that help people think. In: Proceedings of the 14th ACM international conference on Multimedia. 2006 Presented at: Proceedings of the 14th Association for Computing Machinery (ACM) International Conference on Multimedia; Oct 23-27, 2006; Santa Barbara, United States. [CrossRef]
    60. Harper S, Michailidou E, Stevens R. Toward a definition of visual complexity as an implicit measure of cognitive load. ACM Trans Appl Percept 2009 Feb 15;6(2):1-18. [CrossRef]
    61. MacKenzie IS. Fitts' law as a research and design tool in human-computer interaction. Human Comput Interact 2009 Nov 11;7(1):91-139. [CrossRef]
    62. Spillers F. Progressive disclosure. In: The Glossary of Human Computer Interaction. Copenhagen: Interaction Design Foundation; 2016.
    63. Hagberg H, Siebert J, Gervaix A, Daehne P, Lovis C, Manzano S, et al. Improving drugs administration safety in pediatric resuscitation using mobile technology. Stud Health Technol Inform 2016;225:656-657. [Medline]
    64. Cornet VP, Toscos T, Bolchini D, Rohani Ghahari R, Ahmed R, Daley C, et al. Untold stories in user-centered design of mobile health: practical challenges and strategies learned from the design and evaluation of an app for older adults with heart failure. JMIR Mhealth Uhealth 2020 Jul 21;8(7):e17703 [FREE Full text] [CrossRef] [Medline]
    65. Saparamadu AA, Fernando P, Zeng P, Teo H, Goh A, Lee JM, et al. User-centered design process of an mHealth app for health professionals: case study. JMIR Mhealth Uhealth 2021 Mar 26;9(3):e18079 [FREE Full text] [CrossRef] [Medline]
    66. Kortum P, Sorber M. Measuring the usability of mobile applications for phones and tablets. Int J Human Comput Interaction 2015 Jul 31;31(8):518-529. [CrossRef]
    67. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Human Comput Interaction 2008 Jul 30;24(6):574-594. [CrossRef]
    68. Metelmann B, Metelmann C, Schuffert L, Hahnenkamp K, Brinkrolf P. Medical correctness and user friendliness of available apps for cardiopulmonary resuscitation: systematic search combined with guideline adherence and usability evaluation. JMIR Mhealth Uhealth 2018 Nov 06;6(11):e190 [FREE Full text] [CrossRef] [Medline]
    69. Gagnon M, Ngangue P, Payne-Gagnon J, Desmartis M. m-Health adoption by healthcare professionals: a systematic review. J Am Med Inform Assoc 2016 Jan;23(1):212-220 [FREE Full text] [CrossRef] [Medline]
    70. Philippi P, Baumeister H, Apolinário-Hagen J, Ebert DD, Hennemann S, Kott L, et al. Acceptance towards digital health interventions - Model validation and further development of the Unified Theory of Acceptance and Use of Technology. Internet Interv 2021 Dec;26:100459 [FREE Full text] [CrossRef] [Medline]
    71. Bevan N. International standards for HCI and usability. Int J Human Comput Stud 2001 Oct;55(4):533-552 [FREE Full text] [CrossRef]
    72. Dexheimer JW, Kurowski BG, Anders SH, McClanahan N, Wade SL, Babcock L. Usability evaluation of the SMART application for youth with mTBI. Int J Med Inform 2017 Jan;97:163-170 [FREE Full text] [CrossRef] [Medline]
    73. Kushniruk A, Nohr C, Jensen S, Borycki EM. From usability testing to clinical simulations: bringing context into the design and evaluation of usable and safe health information technologies. Yearb Med Inform 2018 Mar 05;22(01):78-85. [CrossRef]
    74. Rowland SP, Fitzgerald JE, Holme T, Powell J, McGregor A. What is the clinical value of mHealth for patients? NPJ Digit Med 2020 Jan 13;3(1):4 [FREE Full text] [CrossRef] [Medline]
    75. Binyamin SS, Zafar BA. Proposing a mobile apps acceptance model for users in the health area: a systematic literature review and meta-analysis. Health Informatics J 2021;27(1):1460458220976737 [FREE Full text] [CrossRef] [Medline]
    76. Vaghefi I, Tulu B. The continued use of mobile health apps: insights from a longitudinal study. JMIR Mhealth Uhealth 2019 Aug 29;7(8):e12983 [FREE Full text] [CrossRef] [Medline]
    77. Lauridsen KG, Nadkarni VM, Berg RA. Man and machine: can apps resuscitate medical performance? Lancet Child Adolescent Health 2019 May;3(5):282-283. [CrossRef]
    78. Safer dosing stronger documentation. Handtevy.   URL: https://www.handtevy.com [accessed 2022-02-20]
    79. Are you sure you're doing the right thing for your patients when it comes to medication delivery? SafeDose.   URL: https://www.safedoseinc.com [accessed 2022-02-20]
    80. Infinite Dose PRO: the smart dosage calculator. Google Play.   URL: https://play.google.com/store/apps/details?id=com.moaness.InfiniteDose.pro&hl=iw&gl=US [accessed 2022-02-20]
    81. Welcome to Pedi QuikCalc!. Pedi QuikCalc.   URL: http://www.pediquikcalc.com/PediQuikCalc/Welcome.html [accessed 2022-02-20]
    82. Higi L, Käser K, Wälti M, Grotzer M, Vonbach P. Description of a clinical decision support tool with integrated dose calculator for paediatrics. Eur J Pediatr 2022 Feb 15;181(2):679-689. [CrossRef] [Medline]
    83. EZDrips homepage. EZDrips.   URL: https://www.ezdrips.com/fr/ [accessed 2022-02-20]
    84. Sezgin E, Özkan-Yildirim S, Yildirim S. Understanding the perception towards using mHealth applications in practice. Inf Development 2016 Dec 19;34(2):182-200. [CrossRef]
    85. Woldeyohannes HO, Ngwenyama OK. Factors influencing acceptance and continued use of mHealth apps. In: HCI in Business, Government and Organizations. Cham: Springer International Publishing; 2017.
    86. Wang C, Qi H. Influencing factors of acceptance and use behavior of mobile health application users: systematic review. Healthcare (Basel) 2021 Mar 22;9(3):357 [FREE Full text] [CrossRef] [Medline]
    87. Assaker G. Age and gender differences in online travel reviews and user-generated-content (UGC) adoption: extending the technology acceptance model (TAM) with credibility theory. J Hospit Market Manag 2019 Aug 28;29(4):428-449. [CrossRef]
    88. Song L, Park B, Oh K. Analysis of the technology acceptance model in examining hospital nurses' behavioral intentions toward the use of bar code medication administration. Comput Inform Nurs 2015 Apr;33(4):157-165. [CrossRef] [Medline]
    89. Timmons S. Nurses resisting information technology. Nurs Inq 2003 Dec;10(4):257-269. [CrossRef] [Medline]
    90. Magsamen-Conrad K, Dillon JM. Mobile technology adoption across the lifespan: a mixed methods investigation to clarify adoption stages, and the influence of diffusion attributes. Comput Human Behav 2020 Nov;112:106456 [FREE Full text] [CrossRef] [Medline]
    91. Hsiao J, Chen R. Understanding determinants of health care professionals' perspectives on mobile health continuance and performance. JMIR Med Inform 2019 Mar 18;7(1):e12350 [FREE Full text] [CrossRef] [Medline]
    92. Akter S, Ray P, D’Ambra J. Continuance of mHealth services at the bottom of the pyramid: the roles of service quality and trust. Electron Markets 2012 May 8;23(1):29-47. [CrossRef]
    93. Holden RJ, Karsh B. The technology acceptance model: its past and its future in health care. J Biomed Inform 2010 Feb;43(1):159-172 [FREE Full text] [CrossRef] [Medline]

    ‎
    CONSORT: Consolidated Standards of Reporting Trials
    CONSORT-EHEALTH: Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online TeleHealth
    EMS: emergency medical service
    HIT: health information technology
    mHealth: mobile health
    OHCA: out-of-hospital cardiac arrest
    PedAMINES: Pediatric Accurate Medication in Emergency Situations
    REDCap: Research Electronic Data Capture
    RN: registered nurse
    SUS: System Usability Scale
    TAM: Technology Acceptance Model
    TTF: task–technology fit
    UTAUT: Unified Theory of Acceptance and Use of Technology

    Edited by A Kushniruk; submitted 03.12.21; peer-reviewed by BJ Nievas-Soriano, M Auerbach; comments to author 24.12.21; revised version received 03.01.22; accepted 20.01.22; published 01.03.22

    Copyright

    ©Johan N Siebert, Laëtitia Gosetto, Manon Sauvage, Laurie Bloudeau, Laurent Suppan, Frédérique Rodieux, Kevin Haddad, Florence Hugon, Alain Gervaix, Christian Lovis, Christophe Combescure, Sergio Manzano, Frederic Ehrler, PedAMINES Trial Group, PedAMINES Prehospital Group. Originally published in JMIR Human Factors (https://humanfactors.jmir.org), 01.03.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on https://humanfactors.jmir.org, as well as this copyright and license information must be included.