This paper is in the following e-collection/theme issue:

Design and Usability of Consumer Health Tech and Home Monitoring Devices (187) Human Factors and Usability Case Studies (739) Usability of Apps and User Perceptions of mHealth (892) Design and Usability of Medical Devices (86) Usability Evaluation Case Studies (182) User-centred Design Case Studies (563) Postoperative Monitoring and Telemonitoring (107)

Published on in Vol 8, No 1 (2021): Jan-Mar

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/24846, first published .
Effects of User Characteristics on the Usability of a Home-Connected Medical Device (Smart Angel) for Ambulatory Monitoring: Usability Study

Effects of User Characteristics on the Usability of a Home-Connected Medical Device (Smart Angel) for Ambulatory Monitoring: Usability Study

Effects of User Characteristics on the Usability of a Home-Connected Medical Device (Smart Angel) for Ambulatory Monitoring: Usability Study

Authors of this article:

Noémie Chaniaud1 Author Orcid Image ;   Olga Megalakaki1 Author Orcid Image ;   Sophie Capo1 Author Orcid Image ;   Emilie Loup-Escande1 Author Orcid Image
Article Authors Cited by (14) Tweetations (3) Metrics

    Original Paper

    Centre de Recherche en Psychologie: Cognition, Psychisme et Organisations, Université de Picardie Jules Verne, Amiens, France

    Corresponding Author:

    Noémie Chaniaud, PhD

    Centre de Recherche en Psychologie: Cognition, Psychisme et Organisations

    Université de Picardie Jules Verne

    Chemin du Thil

    Amiens, UR UPJV 7273

    France

    Phone: 33 3 22 82 70 59

    Email: noemie.chaniaud@u-picardie.fr


    Background: The Smart Angel home medical device allows ambulatory surgery patients to monitor their own health by taking their blood pressure and oxygen levels and answering a health questionnaire from home. Currently, this device is a prototype in the design phase, and no usability evaluation has been performed. This preventive device must be usable by patients with different profiles; however, it is important to select patients carefully to ensure their safety when using the device. As such, it would be interesting to know how to select or exclude patients. However, the links between user characteristics and the usability of this home medical device remain unclear.

    Objective: This study aims to better understand the links between certain characteristics of potential patients (ie, age, education, technophilia, and health literacy) and the usability (ie, effectiveness, efficiency, and satisfaction) of Smart Angel, as defined by the ISO 9241-11.

    Methods: We conducted an experimental study involving 36 participants investigating the effects of 4 patient characteristics (ie, age, education, technophilia, and health literacy) on usability, measured in terms of effectiveness, efficiency, and satisfaction. A mixed methods approach (subjective vs objective) using a variety of standard instruments was adopted (direct observation, video analysis, and questionnaires). First, to help participants project themselves into the real use of the Smart Angel device, they watched a scenario in a video. Second, the participants completed a set of questionnaires to show the extent of their health literacy level (Newest Vital Sign [NVS] and the Health Literacy Survey [HLS]) and then operated Smart Angel devices. Efficiency (ie, handling time) and effectiveness (ie, number of handling errors) measures were collected by video analysis. Satisfaction measures were collected by a questionnaire (System Usability Scale [SUS]). The qualitative observational data were coded using inductive analysis by 2 independent researchers specialized in cognitive psychology and cognitive ergonomics.

    Results: The results show a moderate and positive correlation between age and effectiveness (r=0.359; P=.03) and efficiency (r=0.357; P=.03). There is strong correlation between health literacy scored by the NVS and effectiveness (r=0.417; P=.01), efficiency (r=-0.38; P=.02), and satisfaction (r=0.45; P=.006). However, there is a weak correlation between technophilia and usability and no relationship between education level and usability.

    Conclusions: Our results show that literacy level and age are 2 important factors to consider when selecting future users of the Smart Angel device to ensure patient safety. This study also serves as an example promoting mixed methodologies in assessments of medical device usability that cannot be performed under real-world conditions.

    JMIR Hum Factors 2021;8(1):e24846

    doi:10.2196/24846

    Keywords

    user characteristics; health literacy; home medical devices; usability study; remote health; home health 


    Background

    Outpatient surgery has been on the rise in recent years. Performed operations are increasingly complex and dangerous for patients who have to manage their convalescence at home. The Smart Angel device is a home-connected medical device specifically designed to prevent postsurgical complications related to outpatient surgery. The purpose of this device is to facilitate the patient's return home by maintaining a link with the hospital. Upon returning home after an operation, the patient is required to use the device to send all their vitals 3 times a day for 1 week before returning the equipment to the hospital center. This postoperative follow-up may also enable patients to manage their convalescence better by avoiding all-too-frequent returns to emergency services or outpatient consultations [1].

    Currently, this system is in an early design stage. Like any medical device, this tool must follow safety and quality standards [2] and usability standards [3] to meet the requirements of European Conformity (CE marking) for marketing. However, even today, the deployment of these connected medical devices is still hindered by their complexity of use, directly implying a lack of usability [4-6], thus impacting patient safety. With this in mind, Kortum and Peres commented, “A lack of usability may cost lives” [7].

    Usability

    Usability is defined by the ISO 9241-11 [3] as “the degree to which a product can be used, by identified users, to achieve defined goals in an effective, efficient, and satisfactory manner, within a specified context of use.” This concept, which is still discussed by the scientific community, has 3 distinct dimensions: (1) effectiveness: the accuracy and completeness with which users achieve certain objectives; (2) efficiency: the relationship between accuracy and the resources used to attain it; and (3) satisfaction: user comfort and a positive evaluation of user interaction. Defined by these 3 dimensions, usability is linked to its context of use, characterized by 4 components: the task, the environment, the resources, and the users [3].

    Despite the use of methodologies that involve the user in the design process [8-10], usability problems persist. There are 2 arguments in the literature that may explain this finding: (1) the lack of a standardized framework and method in usability studies [11-15], and (2) a lack of knowledge of the impact of the use context [16,17] on usability, in particular, user characteristics.

    User Characteristics

    Age, Level of Education, Technophilia, and Health Literacy

    Several researchers have recently investigated the relationship between user characteristics and the usability of connected devices in health care [11,18-22]. In particular, 4 user characteristics have been studied in the scientific literature: age [11,20,22-26], level of education [11,19,20], technophilia (ie, experience in information technology and previous experience with medical devices [11,23,27]), and health literacy [20,24,28,29]. In most studies, authors tend to agree on these interrelationships when investigating different devices. We detail these studies below.

    Age

    Many authors have examined the influence of age on the usability of connected devices in health care. Most of these authors concur on the influence of age on usability. For example, Georgsson and Staggers [11] investigated the usability of a diabetes management app running on a smartphone using the metrics of ISO 9241-11 (effectiveness, efficiency, and satisfaction). The authors found that the younger age group (30-49 years old) made fewer errors (ie, was more effective), was faster (ie, more efficient), and more satisfied (System Usability Survey [SUS] score of 88.33 vs 77.14) than the older group (50-69 years old). Sparkes et al [23] examined the usability of remote cardiac testing and found that the age of the participants impacted their ability to install the equipment. Younger subjects appeared to be more comfortable than older subjects. Jones and Caird [25] examined the use of a blood glucose meter and found that younger subjects had fewer difficulties and made fewer errors (ie, were more effective) than older subjects. Mykityshyn et al [26] also examined the use of a glucometer and found that young subjects were faster (ie, more efficient) than older subjects, regardless of the instruction format provided (written and drawn vs video). Van der Vaart et al [20] evaluated the usability of an application for monitoring the symptoms of 32 narcoleptics and found that usability (measured in terms of the number of tasks completed and problems encountered) was moderately and positively correlated with age and eHealth literacy level.

    However, Liang et al [19] found no relationship between age and satisfaction as measured by the SUS score in their study on the evaluation of 7 health devices used by the general public (eg, connected watches), conducted with a sample of 388 participants. Similarly, Jensen et al [18] found no relationship between usability and the age of participants with respect to access and use of online health information. The authors explain that this result is probably due to the contrast in health literacy levels that would have taken precedence over the other variables.

    Level of Education

    The level of education is also a variable found in many usability assessments. However, to our knowledge, no studies have proven this link. Georgsson and Staggers [11], Liang et al [19], and Van der Vaart et al [20] have all found a lack of association between participants' level of education and usability (ie, effectiveness, efficiency, and satisfaction).

    Technophilia

    Differing results have been reported regarding the influence of technophilia—experience with information technologies (IT) and previous experience of medical devices—on usability. Georgsson and Staggers [11] found that those with more technology experience (what the authors call “IT/computer experience”) made fewer errors (ie, were more effective), were faster (ie, more efficient), and were more satisfied with the diabetes management application (+5 points for the SUS score). Conversely, Harte et al [27] conducted regression analyses between technology experience and SUS score and found no significant effect when evaluating a smartphone health app. Finally, Sparkes et al [23] showed that familiarity with the technologies seemed to have an influence on the correct installation of their device.

    Health Literacy
    Definition and Assessments

    Health literacy is a user characteristic that can be expected to influence medical device usability [18,20,28,30]. Due to its multidimensionality, however, this characteristic is complex to define and difficult to assess. Sørensen et al [31] describe it as “an individual's knowledge, skills, motivation, and ability to identify, understand, evaluate, and use health information in decision-making in health care, disease prevention, and health promotion to maintain or improve lifelong quality of life.” However, this notion is often mentioned as a determinant to be considered in therapeutic education [32], prevention [33], therapeutic adherence, access to health information [18], and even recovery rate [32,34]. However, to our knowledge, no study has assessed the level of health literacy among the French population at the national level.

    In terms of evaluation, health literacy is particularly difficult to measure for at least two reasons. The first reason concerns its multidimensional specificity [31]. The second reason is that health literacy is not related to socioeconomic criteria as might be intuitively assumed [35].

    Currently, there are 2 main methods of measuring health literacy [36]: (1) questionnaire methods, by which an individual's abilities are assessed, and (2) self-reported methods, by which an individual's behaviors towards a health professional are directly observed. Currently, few tools exist in the French language compared to the 51 English-language instruments identified by Haun et al [37]. The most frequently used and cited instruments are part of questionnaire-based methods; they are the Test of Functional Health Literacy in Adults (TOFHLA) [38], the Rapid Estimate of Adult Literacy in Medicine (REALM) [39], the European Health Literacy Survey (HLS-EU) [31], and the Newest Vital Sign (NVS) [40]. However, these instruments have several limitations. Among these instruments, the REALM is more like a reading test than a comprehension test since participants are asked to read medical terms. The short version of the TOFHLA (ie, the Short Test of Functional Health Literacy in Adults [S-TOFHLA]), which assesses respondents' level of comprehension, seems more adapted to Swiss culture than to French culture [41] (indeed, direct reference is made to the Swiss health insurance system and the transmission of certain documents that do not apply to the French social security model). In addition, the validity of S-TOFHLA is currently the subject of some controversy due to inconsistencies in the interpretation of its component items [42]. Another instrument proposed in the literature, the NVS [40], shows a strong correlation (Cronbach α>.76) with the measurement of S-TOFHLA [43]. It also assesses some of the respondents' cognitive skills (reading, writing, comprehension, numeracy). Finally, the HLS-EU is based on the multidimensional literacy model of Sørensen et al [31]. This tool has identified important gaps in 8 European countries, as approximately 1 in 2 people reportedly have a problematic or inadequate level of health literacy [44].

    Health Literacy and Usability

    In the context of health technologies such as connected medical devices, which are increasingly becoming part of patient life, studies on the correlation between health literacy and usability are still rare or exploratory. Monkman and Kushniruk [21] propose an assessment of usability by considering health literacy through the design and validation of heuristic criteria. To do so, the authors adapted a set of existing guidelines for designing health-specific websites to make the content more understandable to users with a reliable level of health literacy. Using an electronic personal health record system, Czaja et al [28] were able to show that populations with low literacy levels had more difficulty using these tools. Kim and Xie [29] conducted a systematic review of articles examining the impact of low health literacy on the use of eHealth devices. Based on 74 studies, the authors conclude that the major barrier to accessing and using online health information for individuals with low literacy is strongly related to website usability. Jensen et al [18] found that participants with low levels of health literacy (as measured by REALM) used health technologies less. Those with low levels of health numeracy (as measured by TOFHLA) would have limited access to these technologies. This latter finding is consistent with those of Kaufman et al [24], who also concluded that low numeracy could be a barrier to using a telemedicine system. Chaniaud et al [30] showed that it is necessary to obtain a minimum level of prior health knowledge to use home medical devices. Finally, to our knowledge, no experimental studies have empirically characterized links between health literacy and usability in terms of efficiency, effectiveness, and satisfaction.

    Study Objective

    We have seen that the complexity of using medical devices resides essentially in usability problems [29], all the more so as they must be usable by patients with diverse profiles. In this sense, consideration of user characteristics, including age, education, technophilia, and health literacy, are important factors to consider in the design of a connected medical device such as Smart Angel for a patient's home. However, to our knowledge, no study involving all 4 of these characteristics has been conducted. Moreover, the relation between these characteristics and usability remains unexplored in the literature. Thus, the aim of this paper is to better understand the relationships between the 4 user characteristics of age, educational level, technophilia, and health literacy, and the usability (measured by effectiveness, efficiency, and satisfaction) of a connected medical device intended for a patient's home.

    To do this, we formulated 4 hypotheses: (H1) older users will be less effective, efficient, and satisfied with the Smart Angel connected medical device than younger users [11,20,25,26]; (H2) users with a low level of technophilia (IT and medical device experience) will be less effective, efficient, and satisfied with the Smart Angel connected medical device than those with a high level of technophilia [11,23,27]; (H3) the level of education will not affect the effectiveness, efficiency, and satisfaction with the Smart Angel connected medical device [11,19,20]; and (H4) users with low levels of health literacy (as measured by NVS and HLS-EU scores) will be less effective, efficient, and satisfied with the Smart Angel connected medical device than those with high levels of health literacy [18,24].


    Participants

    We enrolled 36 participants for this study: 17 (47%) females and 19 (53%) males aged 20-64 (mean 40.75, SD 14.45) years. The inclusion criteria were that participants had to (1) have a 4G connection at home, (2) be under 70 years of age, (3) be eligible for outpatient surgery, and (4) not be at home alone. All participants were native French speakers and signed a consent form after being informed of the study's progress. The study was in line with the ethical recommendations of the Declaration of Helsinki. The participants were recruited on a voluntary basis, and no compensation was offered. Handover of the Smart Angel device took place at the participant's home or workplace.

    Materials and Measurements

    The materials for this study included (1) the Smart Angel device, (2) personas and their scenarios, and (3) questionnaires (ie, 2 questionnaires assessing the level of health literacy, namely, the NVS and the HLS-EU; a questionnaire relating to sociodemographic data; and a questionnaire assessing satisfaction, namely, the SUS).

    The Smart Angel Device

    The Smart Angel device is designed by Evolucare Technologies. It consists of a Samsung 9-inch tablet with the Smart Angel application and 2 connected devices, a wrist blood pressure monitor (iHealth BP7) for blood pressure measurement and an oximeter (iHealth Oximeter PO3) for oxygen saturation and pulse measurement, which are available for the general public with European certification (Figure 1). To use the Smart Angel device, it is necessary to access the Smart Angel application and perform a digital medical “appointment” from a tablet application.

    ‎
    Figure 1. The Smart Angel components. Upper left: a pulse oximeter (iHealth Oximeter PO3); lower left: a wrist blood pressure monitor (iHealth BP7); right: a tablet with the Smart Angel application.
    View this figure

    The patient is given step-by-step instructions for connecting to and taking measurements with the blood pressure monitor and the pulse oximeter. The procedures for using the blood pressure monitor and pulse oximeter were built into the application; they include text and images for each step of the operation. For the 2 connected devices, the participant must first have a correct body position to then connect the equipment, install it correctly on themselves, start the measurement, and then remove and switch off the equipment. A schematic representation of this procedure is shown in Figure 2.

    ‎
    Figure 2. Schematic representation of the main steps in the use of the Smart Angel device.
    View this figure

    Once the blood pressure or oxygenation measurement has been taken, the patient's health data are displayed on a colored gauge (from green to orange) according to the level of severity of the constant collected (Figure 3). Then, the user is presented with a questionnaire with various items related to general health, pain, sleep, and nausea. These items are presented either in simple-choice question format (eg, “How are you feeling today? Good, not good, not good at all”) or on a Likert scale (eg, “Rate your pain on a scale of 1 to 10”).

    ‎
    Figure 3. Screenshot of the Smart Angel application (Evolucare Technologies). Left: a form presenting an overview of the subjective state of health; right: the procedure for using the monitor.
    View this figure

    Personas and Scenarios

    We constructed 5 personas and their scenarios based on statistical surveys of outpatient surgical procedure types in France [45] and observations made in the field [46]. Generally used in the design phase, the personas method draws on the theory of mind and the theory of stereotypes and can provoke certain emotional states [47]. The personas scenarios were presented to the participants as audiovisual cartoons. All scenarios were constructed in the same way. Only the type of operation and the cause of the operation changed, according to each persona. An example of a persona is presented in Multimedia Appendix 1.

    Questionnaires

    Measuring Health Literacy: Objective (NVS) and Subjective (HLS-EU) Assessments

    Given the limited options of French-translated and validated health literacy questionnaires, we chose to use 2 health literacy questionnaires for a holistic view of this multidimensional skill: the NVS and the 16-item Health Literacy Survey (HLS-EU-Q16):

    The French-translated [48] NVS [40] is a validated test assessing a patient’s ability to comprehend reading material and manipulate numbers (numeracy). Consequently, the NVS provides an objective assessment of health literacy level. Participants were asked to use an ice cream nutrition label to answer 6 questions (eg, “If I am allergic to peanuts, can I eat this ice cream?” Answer: “No, because the ice cream contains traces of peanut oil”). The total sum of the items (0-6 points) classified respondents into 3 categories: 0-1 point = inadequate health literacy; 2-3 points = problematic health literacy; 4-6 points = sufficient health literacy. The interitem reliability of the NVS in this study was good (Cronbach α=.883) [49].

    The French-translated [50] HLS-EU-Q16 [31] is the short version of the HLS questionnaire. This version is composed of 16 items, 13 of which assess the 4 types of health literacy skills: the ability to access, understand, evaluate, and apply health information. Respondents were asked to rate their own ability to access information (eg, “Please rate, on a scale of very easy to very difficult, how easy is it for you to understand your doctor's or pharmacist's instructions on how to take your medication?”). Consequently, the HLS-EU-Q16 provides a subjective assessment of health literacy level. Answers are provided in 4 categories, on a 4-point Likert scale ranging from ”very easy“ to ”very difficult.“ To calculate the total score, the answers ”easy“ and ”very easy“ were assigned 1 point per item, and the answers ”difficult“ and ”very difficult“ were assigned 0 points per item. The total sum of the items (0-16 points) classified respondents into 3 categories: 0-8 points = inadequate health literacy; 9-12 points = problematic health literacy; 13-16 points = sufficient health literacy. The interitem reliability of the HSL-EU in this study was good (Cronbach α=.803) [49].

    Sociodemographic Measurements (Age, Education Level, Technophilia, Etc)

    This questionnaire includes the following personal details: age, gender, educational level, residential area, technophilia, and hospital experience. IT experience was measured by 2 items, adapted from Agarwal and Prasad [51], related to the participant’s use of and willingness to explore IT innovations (eg, “Which of these technologies do you use and how often?”). On a 5-point Likert scale, the possible answers ranged from “never” to “very often.”

    Measuring Usability (ISO 9241-11:2018)
    Measuring Effectiveness

    Effectiveness was measured by counting the number of manipulation errors, such as not putting the blood pressure cuff in the correct position. With respect to the use of the monitor, 5 categories of errors were identified: the participant (1) did not position the monitor correctly, (2) incorrectly directed the monitor toward the palm of the hand, (3) did not position the forearm correctly, (4) moved during the measurement, or (5) did not connect the monitor's Bluetooth to the tablet. Regarding the use of the pulse oximeter, 4 categories of error were identified: the participant (1) did not position the oximeter the right way, (2) did not insert the finger as far as the sensor, (3) removed the oximeter too early during the measurement, or (4) did not connect the Bluetooth from the oximeter to the tablet. With the tablet, 1 type of error was observed: the participant did not enter the appointment in the application. A scoring grid was used to identify these manipulation errors. When the participant made several attempts, we recorded the cumulative number of errors.

    Measuring Efficiency

    Measuring efficiency was based on the manipulation duration times of the various device tools for 3 measurements: blood pressure monitor manipulation, pulse oximeter manipulation, and total manipulation of the device, including the complete appointment. These times were measured from the time participants first touched the device (monitor, pulse oximeter, or tablet) to the time they turned it off after taking the measurement.

    Measuring Satisfaction

    Satisfaction was measured using the SUS. This ”quick and dirty“ questionnaire [52] consists of 10 items with 5 response options on a Likert scale (ranging from ”strongly disagree“ to ”strongly agree“), which allows for a subjective assessment of usability [53]. We used an adapted and validated version [54], in which we replaced the term ”system“ with the term ”medical device.“ Scores were calculated according to the recommendations of Brooke [52] and ranged from 0 to 100. Lower scores indicate low usability.

    Procedure

    The average duration of this experiment was 45 minutes. The selected participants did not come out of ambulatory surgery. Participants were first invited to choose among 5 proposed personas to allow them to project themselves into the needs of future users of the Smart Angel device [55]. The persona chosen had to be consistent with at least the participant’s age, profession, and previous surgery. Then, the researcher demonstrated the use of the Smart Angel device to the participant for about 3 minutes, sharing information about the correct manipulation of the device (eg, ”The monitor should always be at heart level“). Participants were asked to complete 3 questionnaires: the sociodemographic data questionnaire, the HLS-EU-Q16, and the NVS. Then they were asked to operate the Smart Angel device by taking a blood pressure measurement followed by an oxygen saturation measurement, and finally, by completing the general health questionnaire. There was no time limit for this. The participants were filmed during the process. The researcher could only intervene in the event of a technical problem (eg, battery problem). Finally, after the experiment, the participant had to respond to the SUS.

    Data Analysis

    The videos were analyzed using BORIS (Behavioral Observation Research Interactive Software) [56], which collected data on effectiveness and efficiency. Results were analyzed using SPSS software (version 22; IBM Corp). Each user characteristic was systematically compared to usability components, including effectiveness, efficiency, and satisfaction. For the health literacy measurement, we first analyzed the HLS-EU-Q16 result and then the NVS result. Bivariate correlations, ANOVAs, and Student t tests were performed when the sample met the homoscedasticity criteria, while nonparametric tests (Kruskal-Wallis and Mann-Whitney) were performed when the sample did not meet these criteria.

    Interjudge Reliability: Objective Measures of Effectiveness and Efficiency

    We used intraclass correlation (ICC) to verify interjudge reliability for quantitative data [57]. A 33% double coding of the collected video data was performed. The mean ICC measurement for total manipulation time (efficiency) was 0.978, with a 95% confidence interval of 0.918 to 0.994 (F11,11=45.436; P<.001). The mean ICC measurement (efficiency) for manipulating the monitor was 0.988, with a 95% confidence interval of 0.954 to 0.997 (F11,11=81.635; P<.001). The mean ICC measurement (efficiency) for manipulating the pulse oximeter was 0.956, with a 95% confidence interval of 0.838 to 0.988 (F11,11=22.955; P<.001). The mean ICC measurement (efficiency) for manipulating the tablet was 0.906, with a 95% confidence interval 0.652 to 0.975 (F11,11=10.688; P<.001). The mean measure of the number of manipulation errors (effectiveness) was 0.952, with a 95% confidence interval of 0.842 and 0.985 (F11,11=20.789; P<.001).


    Effects of User Characteristics on Usability

    The correlations between user characteristics and usability components (ie, effectiveness = number of manipulation errors; efficiency = manipulation time; satisfaction = SUS score) were systematically analyzed (Table 1).

    Table 1. Descriptive analyses of user characteristics, user experiences in health, medical devices, and technology (n=36).
    CharacteristicsValueAverage effectiveness, number of errors (SD)Average efficiency, manipulation time in seconds (SD)Average satisfaction, SUS score (SD)
    Age in years, mean (SD)

    		40.75 (14.45)N/AaN/AN/A
    Gender, n (%)

    		Male19 (52.8)1.21 (1.27)362.09 (144.16)87.24 (11.18)


    		Female17 (47.2)2.06 (1.25)373.91 (126.3)81.03 (11.73)
    Education, n (%)

    		Secondary education

    		5 (13.9)

    		2.8 (1.64)

    		337.96 (89.67)77 (9.75)


    		Higher education, 1st cycle11 (30.6)

    		1.36 (1.1)

    		334.99 (106.25)87.05 (13.82)

    		Higher education, 2nd cycle11 (30.6)

    		1.64 (1.2)

    		412.95 (198.6)82.73 (13.34)

    Higher education, 3rd cycle9 (25)1.22 (1.3)368.77 (80.88)86.94 (5.97)
    Residential area, n (%)

    		Rural6 (16.7)1 (0.89)

    		362.92 (76.48)88.75 (6.85)

    		Semi-urban5 (13.9)

    		1.8 (2.05)

    		339.11 (72.3)87 (11.37)

    		Urban25 (64.9)1.72 (1.24)374.52 (154.78)82.7 (12.62)
    Persona chosen, n (%)

    		Persona 18 (22.2)*b**

    		Persona 28 (22.2)***

    		Persona 38 (22.2)***

    		Personal 44 (11.1)***

    		Persona 58 (22.2)***
    Health care experience with operations, n (%)

    		Yes32 (88.9)

    		1.59 (1.21)

    		376.18 (136.24)

    		85,39 (11.72)


    		No4 (11.1)1.75 (2.22)299.55 (106.9)75.62 (8)
    Health care experience with outpatient operations, n (%)

    		Yes18 (50)1.39 (1.33)367.15 (142.09)86.11 (11.8)

    		No18 (50)1.83 (1.29)368.19 (130)82,5 (11.66)
    Health care experience with suffering from a chronic illness, n (%)

    		Yes11 (30.6)1.27 (1.35)386.68 (184.62)81.14 (15.26)

    		No25 (69.4)1.76 (1.3)359.3 (108.77)85.7 (9.8)
    Medical device experience with taking blood pressure, n (%)

    		Yes24 (66.7)1.54 (1.32)361.43 (131.26)63.3 (11.22)

    		No12 (33.3)1.75 (1.36)380.15 (145.06)86.25 (12.9)
    Medical device experience with blood oxygenation testing, n (%)

    		Yes5 (13.9)0.4 (0.55)309.16 (70.87)89 (8.02)


    		No31 (86.1)1.81(1.3)377.11 (140.31)83.56 (12.12)
    Information technology experience with ease of use of tablet/computer/telephone, n (%)

    		Very comfortable23 (63.9)

    		1.35 (1.23)

    		360.16 (120.21)86.85 (11.24)

    		Relatively comfortable11 (30.6)

    		2.27 (1.42)

    		401.34 (166.78)

    		78.18 (11.78)


    		Moderately comfortable2 (5.6)

    		1 (0)

    		268.81 (33.95)88.75 (5.3)


    		Rather uncomfortable0 (0)

    		—c——

    		Not at all comfortable0 (0)—

    		——
    Frequency of use of technology, n (%)

    		Very often (every day)5 (13.9)1 (1.22)

    		314.42 (48.65)

    		92.5 (6.85)


    		Often (several times a week)12 (33.3)

    		1.25 (1.36)

    		364.46 (133.26)

    		90.42 (6.47)


    		Rarely (from time to time)17 (47.2)

    		2 (1.27)

    		364.08 (99.16)79.26 (11.38)

    		Very rarely (occasionally)1 (2.8)

    		3 (—)

    		856.33 (—)

    		55 (—)


    		Never1 (2.8)1 (—)244.8 (—)85 (—)

    aN/A: not applicable.

    b*Highly correlated to the ages of the participants.

    c— Not available.

    Age

    The age of the participants (mean 40.75, SD 14.45, range 20-64 years) is significantly correlated (positively and weakly) with the number of manipulation errors (effectiveness: r=0.359; P=.03) and manipulation time (efficiency: r=0.357; P=.03). On the other hand, there was no significant correlation between age and SUS score (satisfaction: r=-0.138; P=.42). In addition, it is important to note that age is not correlated with the literacy level of the HLS-EU-Q16 (r=0.013; P=.94) or the NVS (r=-0.013; P=.94; Figure 4).

    ‎
    Figure 4. Schematic representation of correlations between age, measurements of participants' health literacy, and usability (ISO 9241-11). HLS-EU-Q16: 16-item European Health Literacy Scale; NVS: Newest Vital Sign. *P<0.5; **P<0.01.
    View this figure
    Technophilia

    The IT experience of participants had no impact on the number of manipulation errors (effectiveness: F5,30=1.229; P=.32) or manipulation time (efficiency: F5,30=1.39; P=.26). On the other hand, there was a significant correlation between IT experience and SUS score (satisfaction: χ(3)=8.671; P=.03).

    Moreover, previous experience of using medical devices that allow users to take their own blood pressure did not influence the number of manipulation errors (effectiveness: t34=0.443; P=.66), the manipulation time (efficiency: t34=0.39; P=.55), or the SUS score (satisfaction: Mann-Whitney U=104; P=.19). Previous experience of using medical devices for taking oxygen levels had a significant effect on the number of manipulation errors (effectiveness: t34=2.359; P=.02; η2=0.14), but this effect was not significant on the manipulation time (efficiency: t34=1.052; P=.30) or the SUS score (satisfaction: t34=-0.965; P=.34).

    Educational Level

    Educational level had no impact on usability in terms of the number of manipulation errors (effectiveness: F3,32=1.889; P=.15), manipulation time (efficiency: F3,32=0.698; P=.56), and SUS score (satisfaction: F3,32=1.076; P=.37).

    Health Literacy

    Systematic analyses were performed comparing the level of literacy (HLS-EU-Q16 and NVS) with each of the components of usability (effectiveness, efficiency, and satisfaction, as per the ISO 9241-11, 2018); Table 2 presents a descriptive representation of the results of the 2 health literacy questionnaires.

    Table 2. Descriptive statistics of the 16-item European Health Literacy Survey (HLS-EU-Q16) and the New Vital Sign (NVS) questionnaires.
    StatisticHLS-EU-Q16NVS
    Mean (SD), range12.97 (2.952), 5-164.17 (2.223), 0-6
    Inadequate health literacy, n (%)3 (8.3)6 (16.7)
    Problematic health literacy, n (%)9 (25)7 (19.4)
    Sufficient health literacy, n (%)24 (66.7)23 (63.9)
    HLS-EU-Q16 Questionnaire Results

    There was no significant correlation between the results of the HLS-EU-Q16 and usability, either in terms of the number of manipulation errors (effectiveness: r=0.34; P=.84), manipulation time (efficiency: r=-0.40; P=.82), or the SUS score (satisfaction: r=0.144; P=.40). After correlation analysis, participants were clustered according to the HLS-EU-Q16 measures (Table 2), following the recommendations of Sørensen et al [31]. No intergroup differences could be observed between the HLS-EU-Q16 results and usability (Table 3) in terms of the number of manipulation errors (effectiveness: F2.33=0.277; P=.76), manipulation time (efficiency: F2.33=0.015; P=.99), and the SUS score (satisfaction: F2.33=0.483; P=.62).

    Table 3. Analyses of the 16-item European Health Literacy Survey (HLS-EU-Q16) score according to usability (effectiveness, efficiency, and satisfaction; n=36).
    HLS-EU-Q16 score classification group (n=36)Effectivenessa, average number of errors (SD)Efficiencyb, average manipulation time (SD)Satisfactionc, SUSd score (SD)
    Inadequate health literacy (n=3)1.67 (2.08)373.26 (88.76)83.33 (3.82)
    Problematic health literacy (n=9)1.89 (1.27)373.35 (98.53)81.11 (14.53)
    Sufficient health literacy (n=24)1.50 (1.28)364.84 (152.73)85.62(11.3)

    aANOVA: F2,33=0.277; P=.76.

    bANOVA: F2,33=0.015; P=.99.

    cANOVA: F2,33=0.483; P=.62.

    dSUS: System Usability Survey.

    NVS Questionnaire Results

    There was a significant mean-size correlation between the results of the French version of the NVS questionnaire and usability (Table 4) in terms of the number of manipulation errors (effectiveness: r=-0.417; P=.01), manipulation time (efficiency: r=-0.38; P=.02), and the SUS score (satisfaction: r=0.45; P=.006). In other words, the higher a participant's level of health literacy (measured using NVS), the fewer manipulation errors they made (ie, they are more effective), the faster they manipulate (ie, they are more efficient), and the higher their SUS score will be (ie, they will be more satisfied).

    After analyzing the correlations, the participants were clustered according to the NVS measurements (Table 2), following recommendations [40]. No intergroup differences could be observed between NVS literacy and usability (Table 4) except for the number of errors (effectiveness: χ2=6.679; P=.04).

    Further intergroup analysis (Figure 4) shows a significant effect between the inadequate-health-literacy and sufficient-health-literacy groups as a function of the number of manipulation errors (effectiveness: Mann-Whitney U=27; P=.02).

    Table 4. Analyses of the New Vital Sign (NVS) results according to usability (effectiveness, efficiency, and satisfaction; n=36).
    NVS score classification group (n=36)Effectivenessa, average number of errors (SD)Efficiencyb, average manipulation time (SD)Satisfactionc, SUSd score (SD)

    Inadequate health literacy
    (n=6)    		2.67 (0.816)463 (165.18)77.08 (14.27)
    Problematic health literacy (n=7)1.71 (0.756)387.72 (219.2)80.71 (15.05)
    Sufficient health literacy (n=23)1.30 (1.43)336.7 (75.79)87.28 (9.07)

    aKruskal-Wallis test: χ2=6.679; P=.035, where P<.05 is significant.

    bKruskal-Wallis test: χ2=3.07; P=.21.

    cANOVA: F2,33=2.392, P=.11; Kruskal-Wallis test: χ2=2.618, P=.27.

    dSUS: System Usability Survey.


    Principal Findings

    This study’s objective was to better understand the relationships between 4 user characteristics (age, education, technophilia, and health literacy) and usability [3] (defined here as effectiveness, efficiency, and satisfaction) with regard to the use of the Smart Angel device. To do this, sociodemographic data were collected, literacy levels were investigated using the HLS-EU-Q16 [31] and the NVS [40], and usability measures were performed (errors and manipulation time, and SUS questionnaire).

    We made 4 hypotheses that age (H1), technophilia (H2), and health literacy (H4) would have an impact on usability, while education level (H3) would not. Our first hypothesis (H1) was that older users would be less effective, efficient, and satisfied with the device compared to younger users. We can partially validate this hypothesis. The results show that the younger the individuals are, the less likely they are to make manipulation errors (ie, they are more effective) and the faster they manipulate the device (ie, they are more efficient). On the other hand, we did not observe any difference between the age of the subjects and the SUS score (satisfaction). All these results are in line with previous research [19,20,25,26]. Indeed, younger users are more effective (eg, Jones and Caird's glucometer [25]) and efficient (eg, Mykityshyn et al's glucometer [26] and Van der Vaart et al's application for narcoleptics [20]) compared to older users, with a positive and medium correlation [20]. However, younger users are as satisfied (SUS score) with the device as older users, which is consistent with the findings of Liang et al [19] while at variance with those of Georgsson and Staggers [11].

    Our second hypothesis (H2) focused on technophilia (experience of information technology and medical devices). The results provide partial validation of this hypothesis, as no correlation was observed between IT experience and usability in terms of effectiveness and efficiency. On the other hand, the technophile participants had a significantly better SUS score (satisfaction) than participants with a low level of technophilia. While these results are consistent with those of Harte et al [27], they contradict previous works [11,23]. We explain these results by a relatively homogeneous representation of IT experience as a function of the age of participants in our sample. We believe that these items [51] highlight the subjective representation of technology use (in relation to age) rather than actual performance in the use of hardware. It is possible that older people may feel that they can properly manipulate a tablet without using other features available in the tool. They would then consider themselves to be quite technophilic, as they would be effective in the day-to-day use of the technology. However, their real capacity to adapt to the technologies is unknown. For example, if an update were to be performed on one of the applications commonly used, it is possible that this would destabilize the manipulation carried out by these individuals.

    We also observed a correlation between experience with medical devices and usability. However, previous experience in the use of a blood pressure monitor had no impact on usability. Conversely, previous experience in the use of a pulse oximeter had a significant effect on effectiveness. Participants who had previously manipulated a pulse oximeter made significantly fewer errors than those who had never manipulated a pulse oximeter. In contrast, previous experience using a pulse oximeter had no effect on efficiency and satisfaction. All subjects who reported previous use of a pulse oximeter also reported previous manipulation of a blood pressure monitor. This result suggests that prior use of a pulse oximeter in combination with a blood pressure monitor would facilitate manipulation of the Smart Angel device in terms of effectiveness. We believe that participants who are accustomed to using this type of complex device are accustomed to being involved in health issues, which may be evidence of strong patient involvement in their own health [58].

    Our third hypothesis (H3) was concerned with the lack of correlation between education level and usability. The results supported our hypothesis, as no significant correlation was found between participants' level of education and usability in terms of effectiveness, efficiency, and satisfaction. These results are also consistent with previous works [11,19,20].

    Finally, the fourth hypothesis (H4) postulated that health literacy influences usability (effectiveness, efficiency, and satisfaction). The HLS-EU-Q16 scores showed no effect on usability (Figure 4). In contrast, the NVS scores showed a significant effect on the number of manipulation errors (effectiveness), manipulation time (efficiency), and SUS score (satisfaction). This is consistent with the results of previous studies [18,28,29]. Significant and medium-sized correlations between the NVS score and each of the usability dimensions were observed (Figure 4). This suggests that the higher the literacy level of the participants, the fewer manipulation errors they make (ie, the more effective they are), the faster they are (ie, the more efficient they are), and the higher the SUS score will be (ie, the more satisfied they are). However, after clustering the participants as recommended [40], there is a significant correlation between NVS literacy level and the number of errors (effectiveness) but no correlation with the manipulation time (efficiency) and the SUS score (satisfaction). Participants with a sufficient literacy level made significantly fewer errors than those with inadequate or problematic literacy.

    It is important to note that the HLS-EU and NVS results are contradictory and demonstrate the complexity of health literacy assessment. In addition, our results suggest that the HLS-EU questioning the participants’ subjective abilities to access health information and make decisions introduces a significant bias in the measurement of health literacy. Some participants may claim to have no difficulty using health information, but there is no verification that this is, in fact, the case. Conversely, the NVS instrument appears to be better suited to gathering information on subjects' cognitive abilities, as it is a test that collects information on participants' thought processes when reading a food label, thus providing a more objective assessment of health literacy.

    Conclusions and Research Prospects

    This study provides theoretical insight into the effects of user characteristics (eg, age, experience, education, and health literacy) through the use of personas with respect to usability (effectiveness, efficiency, and satisfaction, according to ISO 9241-11 [3]) in the case of the Smart Angel connected medical device. This study provides a methodological contribution insofar as it revealed the differences in data collection between the NVS and the HLS-EU-Q16, thus demonstrating the importance of continuing research in the field of health literacy measurement tools. In addition, these results allow us to better understand the importance of the impact of technophilia among older people with a sufficient level of health literacy for usability.

    The results of this study suggest 4 research prospects. First, the relevance of the personas method in the prototype evaluation phase has never been proven. This method is classically used in the design phase by designers (ergonomists, designers, engineers, and even future users) but more rarely used in an evaluation framework. To validate this method in this new context of use in the evaluation phase, it would be necessary to reproduce this study by adding a control group (ie, a group for whom the personas are not presented). Secondly, the training carried out by the researcher could be adapted according to the literacy levels of the participants. Indeed, the main difficulty in the use of a medical device is understanding the procedures, and this cannot be achieved if there is insufficient upstream training [59]. Training should certainly be adapted to the ages and literacy levels of the participants. Demonstration by the researcher may be sufficient for groups with adequate levels of health literacy. Conversely, for groups with inadequate or problematic levels of health literacy, further instruction should be considered. Third, the choice of questionnaire is a crucial step in measuring health literacy. Indeed, we observed a significant disparity in results between the HLS-EU-Q16 and the NVS. As already discussed, these 2 questionnaires do not appear to assess the same dimensions of health literacy. Further work is needed to understand what exactly is being assessed by each of the health literacy questionnaires. We believe that it is better to evaluate this skill with objective assessments. In the same way, it would have been interesting to perform objective measurements of technophilia.

    Finally, beyond health literacy, it would now be appropriate to measure the level of eHealth literacy [20]. Unfortunately, there is no valid questionnaire in French on this subject. Thus, more systematic translations and adaptations of these tools should be considered in future studies.

    Currently, as a result of this study, the Smart Angel device is in clinical trials where usability tests continue to be carried out in in situ conditions.

    Acknowledgments

    This project was financially supported by the European Regional Development Fund (ERDF), Evolucare, and the Programme d'Investissements d'Avenir (PIA), which is one of 5 research and development projects structuring competitiveness of the CSPP.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Example of a persona.

    PDF File (Adobe PDF File), 707 KB
    1. Ghosh S, Sallam S. Patient satisfaction and postoperative demands on hospital and community services after day surgery. Br J Surg 1994 Nov;81(11):1635-1638. [CrossRef] [Medline]
    2. International Organization for Standardization (ISO). Medical devices - Application of risk management to medical devices. 2019.   URL: https://www.iso.org/standard/72704.html [accessed 2019-03-03] [WebCite Cache]
    3. International Organization for Standardization (ISO). Ergonomics of human-system interaction - Part 11: Usability: Definitions concepts. 2018.   URL: https://www.iso.org/standard/63500.html [accessed 2018-03-03] [WebCite Cache]
    4. Carayon P, Hoonakker P. Human Factors and Usability for Health Information Technology: Old and New Challenges. Yearb Med Inform 2019 Aug;28(1):71-77 [FREE Full text] [CrossRef] [Medline]
    5. Cifter AS. Blood Pressure Monitor Usability Problems Detected Through Human Factors Evaluation. Ergonomics in Design 2017 May 22;25(3):11-19. [CrossRef]
    6. Greysen SR, Chin Garcia C, Sudore RL, Cenzer IS, Covinsky KE. Functional impairment and Internet use among older adults: implications for meaningful use of patient portals. JAMA Intern Med 2014 Jul 01;174(7):1188-1190 [FREE Full text] [CrossRef] [Medline]
    7. Kortum P, Peres SC. Evaluation of Home Health Care Devices: Remote Usability Assessment. JMIR Hum Factors 2015 Jun 05;2(1):e10 [FREE Full text] [CrossRef] [Medline]
    8. LeRouge C, Ma J, Sneha S, Tolle K. User profiles and personas in the design and development of consumer health technologies. Int J Med Inform 2013 Nov;82(11):e251-e268. [CrossRef] [Medline]
    9. Matthew-Maich N, Harris L, Ploeg J, Markle-Reid M, Valaitis R, Ibrahim S, et al. Designing, Implementing, and Evaluating Mobile Health Technologies for Managing Chronic Conditions in Older Adults: A Scoping Review. JMIR Mhealth Uhealth 2016 Jun 09;4(2):e29 [FREE Full text] [CrossRef] [Medline]
    10. Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, et al. A user-centered model for designing consumer mobile health (mHealth) applications (apps). J Biomed Inform 2016 Apr;60:243-251 [FREE Full text] [CrossRef] [Medline]
    11. Georgsson M, Staggers N. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. J Am Med Inform Assoc 2016 Jan;23(1):5-11 [FREE Full text] [CrossRef] [Medline]
    12. Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009 May;78(5):340-353. [CrossRef] [Medline]
    13. Nelson LA, Mulvaney SA, Johnson KB, Osborn CY. mHealth Intervention Elements and User Characteristics Determine Utility: A Mixed-Methods Analysis. Diabetes Technol Ther 2017 Jan;19(1):9-17 [FREE Full text] [CrossRef] [Medline]
    14. Peute LW, Driest KF, Marcilly R, Bras Da Costa S, Beuscart-Zephir M, Jaspers MWM. A framework for reporting on human factor/usability studies of health information technologies. Stud Health Technol Inform 2013;194:54-60. [Medline]
    15. Yen P, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012;19(3):413-422 [FREE Full text] [CrossRef] [Medline]
    16. Borsci S, Uchegbu I, Buckle P, Ni Z, Walne S, Hanna GB. Designing medical technology for resilience: integrating health economics and human factors approaches. Expert Rev Med Devices 2018 Jan 20;15(1):15-26. [CrossRef] [Medline]
    17. Grebin SZ, Echeveste MES, Magnago PF, Tanure RLZ, Pulgati FH. [Analytical strategy for assessment of usability of medical devices from the user's perspective: a study of hemodialysis patients]. Cad Saude Publica 2018 Aug 20;34(8):e00074417 [FREE Full text] [CrossRef] [Medline]
    18. Jensen JD, King AJ, Davis LA, Guntzviller LM. Utilization of internet technology by low-income adults: the role of health literacy, health numeracy, and computer assistance. J Aging Health 2010 Sep;22(6):804-826. [CrossRef] [Medline]
    19. Liang J, Xian D, Liu X, Fu J, Zhang X, Tang B, et al. Usability Study of Mainstream Wearable Fitness Devices: Feature Analysis and System Usability Scale Evaluation. JMIR Mhealth Uhealth 2018 Nov 08;6(11):e11066 [FREE Full text] [CrossRef] [Medline]
    20. van der Vaart R, van Driel D, Pronk K, Paulussen S, Te Boekhorst S, Rosmalen JGM, et al. The Role of Age, Education, and Digital Health Literacy in the Usability of Internet-Based Cognitive Behavioral Therapy for Chronic Pain: Mixed Methods Study. JMIR Form Res 2019 Nov 21;3(4):e12883 [FREE Full text] [CrossRef] [Medline]
    21. Monkman H, Kushniruk A. A health literacy and usability heuristic evaluation of a mobile consumer health application. Stud Health Technol Inform 2013;192:724-728. [Medline]
    22. Sonderegger A, Schmutz S, Sauer J. The influence of age in usability testing. Appl Ergon 2016 Jan;52:291-300. [CrossRef] [Medline]
    23. Sparkes J, Valaitis R, McKibbon A. A usability study of patients setting up a cardiac event loop recorder and BlackBerry gateway for remote monitoring at home. Telemed J E Health 2012;18(6):484-490. [CrossRef] [Medline]
    24. Kaufman DR, Patel VL, Hilliman C, Morin PC, Pevzner J, Weinstock RS, et al. Usability in the real world: assessing medical information technologies in patients' homes. J Biomed Inform 2003;36(1-2):45-60 [FREE Full text] [CrossRef] [Medline]
    25. Jones J, Caird JK. The Usability of Blood Glucose Meters: Task Performance Differences Between Younger and Older Age Groups. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2017 Sep 28 Presented at: Proceedings of the Human Factors and Ergonomics Society Annual Meeting Internet; 2017 Sep 1; 61st International Annual Meeting, JW Marriott, Austin, Texas, USA p. 604-608. [CrossRef]
    26. Mykityshyn AL, Fisk AD, Rogers WA. Learning to use a home medical device: mediating age-related differences with training. Hum Factors 2002;44(3):354-364. [CrossRef] [Medline]
    27. Harte R, Hall T, Glynn L, Rodríguez-Molinero A, Scharf T, Quinlan LR, et al. Enhancing Home Health Mobile Phone App Usability Through General Smartphone Training: Usability and Learnability Case Study. JMIR Hum Factors 2018 Apr 26;5(2):e18 [FREE Full text] [CrossRef] [Medline]
    28. Czaja SJ, Zarcadoolas C, Vaughon WL, Lee CC, Rockoff ML, Levy J. The usability of electronic personal health record systems for an underserved adult population. Hum Factors 2015 May 05;57(3):491-506 [FREE Full text] [CrossRef] [Medline]
    29. Kim H, Xie B. Health literacy in the eHealth era: A systematic review of the literature. Patient Educ Couns 2017 Jun;100(6):1073-1082. [CrossRef] [Medline]
    30. Chaniaud N, Métayer N, Megalakaki O, Loup-Escande E. Effect of Prior Health Knowledge on the Usability of Two Home Medical Devices: Usability Study. JMIR Mhealth Uhealth 2020 Sep 21;8(9):e17983 [FREE Full text] [CrossRef] [Medline]
    31. Sørensen K, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, (HLS-EU) Consortium Health Literacy Project European. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health 2012 Jan 25;12(1):80 [FREE Full text] [CrossRef] [Medline]
    32. Williams MV, Baker DW, Parker RM, Nurss JR. Relationship of functional health literacy to patients' knowledge of their chronic disease. A study of patients with hypertension and diabetes. Arch Intern Med 1998 Jan 26;158(2):166-172. [CrossRef] [Medline]
    33. Mõttus R, Johnson W, Murray C, Wolf MS, Starr JM, Deary IJ. Towards understanding the links between health literacy and physical health. Health Psychol 2014 Feb;33(2):164-173. [CrossRef] [Medline]
    34. Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med 2011 Jul 19;155(2):97-107. [CrossRef] [Medline]
    35. Duteille F. Compétences en santé déficientes : obstacle à une prise en charge optimale. Revue Médicale Suisse, 8 2019 Apr:1016-1021 [FREE Full text] [CrossRef]
    36. Jordan JE, Osborne RH, Buchbinder R. Critical appraisal of health literacy indices revealed variable underlying constructs, narrow content and psychometric weaknesses. J Clin Epidemiol 2011 Apr;64(4):366-379. [CrossRef] [Medline]
    37. Haun JN, Valerio MA, McCormack LA, Sørensen K, Paasche-Orlow MK. Health literacy measurement: an inventory and descriptive summary of 51 instruments. J Health Commun 2014 Oct 14;19 Suppl 2(sup2):302-333. [CrossRef] [Medline]
    38. Parker R. Health literacy: a challenge for American patients and their health care providers. Health Promot Int 2000;15(4):277-283 [FREE Full text] [CrossRef]
    39. Murphy P, Davis T, Long S, Jackson R, Decker B. Rapid Estimate of Adult Literacy in Medicine (REALM): A Quick Reading Test for Patients. Journal of Reading 1993;37(2):124-130 [FREE Full text]
    40. Weiss BD, Mays MZ, Martz W, Castro KM, DeWalt DA, Pignone MP, et al. Quick assessment of literacy in primary care: the newest vital sign. Ann Fam Med 2005;3(6):514-522 [FREE Full text] [CrossRef] [Medline]
    41. Connor M, Mantwill S, Schulz PJ. Functional health literacy in Switzerland--validation of a German, Italian, and French health literacy test. Patient Educ Couns 2013 Jan;90(1):12-17. [CrossRef] [Medline]
    42. Housten AJ, Lowenstein LM, Hoover DS, Leal VB, Kamath GR, Volk RJ. Limitations of the S-TOFHLA in measuring poor numeracy: a cross-sectional study. BMC Public Health 2018 Mar 27;18(1):405 [FREE Full text] [CrossRef] [Medline]
    43. Baker DW, Williams MV, Parker RM, Gazmararian JA, Nurss J. Development of a brief test to measure functional health literacy. Patient Educ Couns 1999 Sep;38(1):33-42. [CrossRef] [Medline]
    44. Sørensen K, Pelikan JM, Röthlin F, Ganahl K, Slonska Z, Doyle G, HLS-EU Consortium. Health literacy in Europe: comparative results of the European health literacy survey (HLS-EU). Eur J Public Health 2015 Dec;25(6):1053-1058 [FREE Full text] [CrossRef] [Medline]
    45. Agence Technique De L'Information sure L'Hospitalisation (ATIH). Etat des lieux 2015 sur l'activité de chirurgie ambulatoire. 2016.   URL: https://www.atih.sante.fr/sites/default/files/public/content/1504/note_gdr_chirambu_2015.pdf [accessed 2016-03-03] [WebCite Cache]
    46. Loup-Escande E, Valentin A, Metayer N, Ricquebourd V, Bidard A, Damay P, et al. Adaptation de la démarche ergonomique en phase amont de la conception de technologies e-santé?xemple du projet Smart Angel. 2019 Presented at: EPIQUE; 9 au 12 juillet; Lyon p. e.
    47. Lallemand C, Gronier G. Methodes de design UX: 30 méthodes fondamentales pour concevoir et évaluer les systèmes interactifs. In: Methodes de design UX: 30 méthodes fondamentales pour concevoir et évaluer les systèmes interactifs. Paris: Editions Eyrolles; 2018:712.
    48. Mansfield ED, Wahba R, Gillis DE, Weiss BD, L'Abbé M. Canadian adaptation of the Newest Vital Sign©, a health literacy assessment tool. Public Health Nutr 2018 Aug;21(11):2038-2045. [CrossRef] [Medline]
    49. Nunnaly J. Psychometric Theory. Chicago: McGraw Hill; 1978.
    50. Rouquette A, Nadot T, Labitrie P, Van den Broucke S, Mancini J, Rigal L, et al. Validity and measurement invariance across sex, age, and education level of the French short versions of the European Health Literacy Survey Questionnaire. PLoS One 2018 Dec 6;13(12):e0208091 [FREE Full text] [CrossRef] [Medline]
    51. Agarwal R, Prasad J. A Conceptual and Operational Definition of Personal Innovativeness in the Domain of Information Technology. Information Systems Research 1998 Jun;9(2):204-215. [CrossRef]
    52. Brooke J. SUS - A quick and dirty usability scale. In: Jordan P, Thomas B, McClelland I, Weerdmeester B, editors. Usability evaluation in industry. London: CRC Press; 1996:252.
    53. Lewis J, Sauro J. The Factor Structure of the System Usability Scale. In: Human Centered Design Internet Springer, Berlin, Heidelberg. 2009 Presented at: Interantion Conference on Human Centered Design; 2009; Berlin p. 94-103. [CrossRef]
    54. Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction 2008 Jul 30;24(6):574-594. [CrossRef]
    55. Bornet C, Brangier E. La méthode des personas: principes, intérêts et limites. Bulletin de psychologie Internet 2013 May 3;524:115-134 [FREE Full text] [CrossRef]
    56. Friard O, Gamba M. : a free, versatile open‐source event‐logging software for video/audio coding and live observations. Methods Ecol Evol 2016 May 28;7(11):1325-1330. [CrossRef]
    57. Elie C, Colombet I. Methods of evaluation of reproducibility. Sang thrombose vaisseaux---Sang Thrombose Vaisseaux Internet 2011 Mar 2011;23(3):138-145. [CrossRef]
    58. Greene J, Hibbard JH. Why does patient activation matter? An examination of the relationships between patient activation and health-related outcomes. J Gen Intern Med 2012 May 30;27(5):520-526 [FREE Full text] [CrossRef] [Medline]
    59. Larsen T. Nurses' instruction of patients in the use of INR-monitors for self-management of cardio-vascular conditions: Missed instructional opportunities. Patient Educ Couns 2017 Apr;100(4):673-681. [CrossRef] [Medline]

    ‎
    HLS-EU: European Health Literacy Survey
    HLS-EU-Q16: 16-item European Health Literacy Survey
    ICC: intraclass correlation
    IT: information technology
    NVS: Newest Vital Sign
    REALM: Rapid Estimate of Adult Literacy in Medicine
    SUS: System Usability Survey
    S-TOFHLA: Short Test of Functional Health Literacy in Adults
    TOFHLA: Test of Functional Health Literacy in Adults

    Edited by A Kushniruk; submitted 07.10.20; peer-reviewed by D Michelet, D Chrimes; comments to author 23.12.20; revised version received 28.12.20; accepted 03.01.21; published 17.03.21

    Copyright

    ©Noémie Chaniaud, Olga Megalakaki, Sophie Capo, Emilie Loup-Escande. Originally published in JMIR Human Factors (http://humanfactors.jmir.org), 17.03.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on http://humanfactors.jmir.org, as well as this copyright and license information must be included.