A Technological Tool Aimed at Self-Care in Patients With Multimorbidity: Cross-Sectional Usability Study

Introduction

Multimorbidity, which is generally defined as the presence of two or more simultaneous chronic diseases in a patient, is a major challenge for health systems [1]. In the European Union, up to 50 million people are estimated to have multimorbidity [2]. Barnett et al [3] estimated a multimorbidity prevalence of 64.9% among patients aged 65-84 years and of 81.5% for those 85 years or older [3]. In recent years, patient-centered care models [4] and, more specifically, interventions aimed at self-care education have made it possible to optimize how patients with multimorbidity manage their chronic diseases [5]. This type of intervention makes it easier for patients to identify their health-problem needs and to identify techniques that can help them make decisions, take appropriate actions, and modify them as they present changes in their diseases [6].

Information and communication technologies (ICTs) have been positioned as useful tools to facilitate self-care [7]. The different self-care strategies using ICTs include those dedicated to the monitoring of biometric parameters through wearable technologies or portable devices and mobile apps [8,9]. To achieve positive health outcomes from these interventions, the interaction between a patient and technology is particularly important. The description of this interaction between the technology, the specific tasks to be developed, and the end user is a property known as usability [10].

Research on usability has grown in parallel with the development of ICTs in health [11]. Reports from international organizations such as the 2012-2020 eHealth Action Plan of the European Commission [12] and the World Health Organization Global Strategy on Digital Health 2020-2025 [13] summarized the importance of the development of technological tools that take into account their interaction with the special conditions of older adults. The study of usability can help determine the reasons for low patient adherence and adoption of a specific technological tool. The improvements in usability could facilitate interaction through several mechanisms: reducing anxiety related to the use of new tools, increasing accessibility and distribution among a greater number of users, and reducing the possible risks derived from misuse [10].

In evaluating usability, the International Organization for Standardization (ISO) 9126 standard [14] assesses the quality of the product [15] and the ISO 9241 standard focuses on processes, referred to as “the extent to which users in a specific environment can use a product to achieve objectives of effectiveness, efficiency and satisfaction in a particular task” [16]. Thus, usability comprises the effectiveness in usability, defined as the degree of completion [11,17,18] and the total number of errors per task; efficiency in usability, defined as the average time to perform the task; and satisfaction in usability, defined as the degree to which the user’s physical and emotional responses resulting from the use of a product satisfy their needs and expectations. The most commonly used methods for usability evaluation are questionnaires and interviews carried out after the use of the technological tool for a certain period of time.

The systematic reviews of Saeed et al [17] and Zapata et al [18] analyzed the definitions of the ISO standards and the methods used in evaluating the usability of health-related technological tools. Their results indicate that the most frequent usability problems are those related to visual aspects of the system and the ability to learn and use specific features [17,18]. However, because the results are limited to a specific technology and may not be generalizable, their interpretation should take into account the special characteristics of the end users, including their health conditions.

The specific characteristics of patients with chronic diseases can affect their interaction with different technologies. For example, in the usability evaluation studies of an app for diabetes self-management [19,20] and that conducted on an automatic drug dispenser for patients with dementia [21], characteristics of the patients were identified that interact with different aspects of usability. Relatedly, Wildenbos et al [22] differentiated four traits related to aging and chronic diseases that act as barriers to usability: cognitive, physical, motivational, and perceptual. Although the development of technological tools aimed at self-care is increasing, their usability has thus far mainly been evaluated in patients with specific isolated pathologies such as in the previous examples. Research from the perspective of patients with multimorbidity has been increasing in recent years [23,24] but remains insufficient [25], even though multimorbidity is the most common way of reporting chronic diseases in the population over 60 years of age [3].

Thus, studying the usability of ICTs has become a key element in the field of multimorbidity [26] to ensure its relevant role in promoting self-care [7]. Along these lines, the TeNDER (Affective Based Integrated Care for Better Quality of Life) project [27,28] was a multisectoral project funded by Horizon 2020, the EU Framework Programme for research and innovation. The TeNDER project developed an integrated care model to manage multimorbidity in patients with dementia, Parkinson disease, and cardiovascular disease in four European countries: Spain, Germany, Italy, and Slovenia. One of the clinical studies related to the TeNDER project was a multicenter, randomized, parallel-group clinical trial carried out in Spain with the main objective of evaluating the effectiveness of the TeNDER system to improve quality of life in patients with chronic diseases. Secondary aims were to describe the satisfaction of patients and their caregivers and the usability of the TeNDER system [29].

The objective of this study was to analyze the usability of a technological tool (TeNDER) dedicated to health and self-care in patients with multimorbidity in primary care.

Methods

Design

This was a descriptive observational cross-sectional study of usability. This study was framed in the clinical trial in the primary care health centers of Madrid Health Service of the TeNDER project (ClinicalTrials.gov NCT05681065) [29].

Ethical Considerations

This study respects the basic ethical principles of autonomy, beneficence, justice, and nonmaleficence, and its development followed the norms of Good Clinical Practice and the principles enunciated in the latest Declaration of Helsinki (Seoul 2013). The study obtained a favorable report from the Research Ethics Board of the Hospital Universitario 12 de Octubre (20/450) and was approved by the Central Research Commission of the Community of Madrid (PC:39/20). Informed consent was obtained from all participants involved in the study. No camera recording or any other identification was made. Patients were included with an anonymous identifier in the data collection logbook (DCL). All data were processed based on the provisions of the EU General Data Protection Regulation 2016/679 of the European Parliament and the Council (April 27, 2016) and the Organic Law on Data Protection and Guarantee of Digital Rights in the Spanish territory (LOPDGDD 3/2018 of 5 December). Participants did not receive any financial compensation for their participation in the study. The only compensation was that received through the user experience during the use of the technological tool.

Population and Sample

The study population included patients with one or more chronic diseases recruited from four primary care health centers in the Community of Madrid that had been included in the TeNDER project by their referring professionals [27,28].

Patients over 60 years of age who had visited their health center in the last year and who had any of the following chronic diseases were included: mild-moderate cognitive impairment (Mini-Mental State Evaluation [MMSE] score 19-28 points), Parkinson disease, or cardiovascular disease, which includes patients with heart failure, chronic ischemic heart disease, or atrial fibrillation. Patients with a life expectancy of less than 6 months based on the opinion of their health care professionals, severe mental illness, incapacity for autonomous movement, or an MMSE score of less than 19 points were excluded.

The 250 patients included in the primary care health centers of Madrid Health Service for the TeNDER project were invited to participate via text messages with a mobile instant messaging app. Thirty-eight patients with multimorbidity (≥2 chronic diseases) agreed to participate. Considering an approximate 90% completion rate of tasks in previous usability evaluation studies of monitoring tools [19,25,30], with this sample size, we report a precision of 9.6% with the 95% CI.

TeNDER Technological Tool

The TeNDER technological tool is a web-based platform that included integrating sensors such as a smartwatch for monitoring physical activity, a sleep tracker to study sleep activity, and a mobile app in which the data collected are displayed and tools for self-management are offered (Figure 1). All of the TeNDER ecosystem technology was developed through a co-design process with all relevant stakeholders using a patient-centered approach. During the project, the functionalities and the mobile app were validated and released after user validation within an incremental development approach, ensuring a feedback framework that provided iterative refinement and improvements of the mobile app.

‎

Variables

The main variable of the study was usability effectiveness, which was determined by the degree of completion and the total number of errors per task. Five tasks were designed using the TeNDER system (Textbox 1). The tasks to be evaluated were based on real case scenarios to simulate how patients would interact with the system in a real-life situation according to the care and self-management process based on the main functionalities of the TeNDER app [29]. The tasks were validated by a panel of three health professionals with experience in the study of usability to verify the accuracy of the content and the context. The degree of completion of the tasks was coded using three categories: (0) not completed when the subject was unable to complete the task (inability to progress or to request advanced help or interruption in task execution), (1) completed with personalized help in the form of comments or directed indications, and (2) completed independently when the user was able to carry out the task either without any help from the person in charge of the test or with the aid of minor indications. An error was coded when the subject made errors that could not be solved or that prevented further progress.

As secondary variables, usability efficiency was determined by timing each individual task and calculating the average time in each task. Usability satisfaction was quantified by administering the System Usability Scale (SUS) (Multimedia Appendix 1) in its Spanish-validated version [31]. For this scale, the global score ranges from 0 to 100, where higher values indicate greater usability satisfaction. According to Bangor et al [32], SUS scores of 70-100 indicate acceptable, whereas scores of 0-50 indicate not acceptable; scores between 50 and 70 are considered to indicate marginally acceptable results.

Sociodemographic variables collected included age, sex, and education level, and clinical variables included type and number of chronic diseases. Chronic pathologies were identified according to the proposals in the O’Halloran classification [33] (Multimedia Appendix 2).

Technology-related variables included previous use of touch screens and the affinity for technology interaction (ATI) scale [34]. For this scale, the global score ranges between 1 and 6, where higher values indicate a greater affinity for the technology (Multimedia Appendix 3).

Data Collection

The variables were collected by interview with the patient in consultation with their referring professional and were recorded in an electronic DCL designed ad hoc with the Research Electronic Data Capture (REDCap) tool hosted on the secure storage server of the institution. REDCap is a secure, web-based software platform designed to support data capture for research studies [35,36].

The patients received the TeNDER technological tool. The usability study was carried out 48 hours afterward based on the execution of tasks in a face-to-face session with a member of the research team who could provide assistance. To record the variables that measure usability, a real-time screen recording of the mobile device was performed during the entirety of task performance. One member of the research team analyzed the recordings. The start and end times were determined from the time the instructions were offered until the moment each task was completed; that information was subsequently transferred to the DCL.

Statistical Analysis

The categorical variables are described as frequencies and percentages. The quantitative variables are described as medians and IQR, as they were nonnormally distributed for the number of patients under study. The main result variable was the proportion of completed tasks (usability effectiveness) with its 95% CI. As secondary outcome variables, the mean effective time to perform each of the tasks (usability efficiency) and the mean score in the SUS questionnaire (usability satisfaction) were estimated. The association of the different usability components (efficacy, effectiveness, and satisfaction) with the sociodemographic and clinical variables was evaluated using the χ² test for categorical variables and the Student t test for quantitative variables (the Mann-Whitney U test was used for comparison of variables that did not follow a normal distribution). The factors associated with completing the task in an independent manner were analyzed using a multiple logistic regression model with robust estimators. The dependent variable was completing the task autonomously. The independent variables were those found to be statistically significant in the bivariable analyses or variables that are otherwise considered to be clinically important. STATA 14 software was used for all statistical analyses.

Results

Among the 250 patients included in the TeNDER project invited, 38 (15.2%) agreed to participate in this study (Figure 2). There were no differences in sociodemographic characteristics between those who refused to participate and the final sample. Finally, 30 patients completed the usability study.

‎

The median age of the participants was 75 (IQR 72.0-79.0) years and 20/38 (52.6%) were women. With a median of 8 (IQR 7.0-11.0) chronic diseases, 89.5% of the patients had at least one cardiovascular risk factor. Syndromes that include anxiety and depression occurred at significantly different rates between women (11/20, 55%) and men (3/18, 16.7%). A total of 83.8% of the patients had previously interacted with touch screens, and the median result on the ATI scale was 3.4 (IQR 3.0-3.8), which differed between men and women.

The baseline characteristics of the patients are presented in Table 1.

Thirty patients completed the usability study. Among them, 66.7% (20/30) completed all the tasks to be evaluated. All patients were able to complete at least three of the five proposed tasks and 10 patients did not complete at least one task. At least one mistake was made while carrying out the tasks in 28/30 patients. A total of 66.7% (10/15) of the women required personalized help in at least one of the tasks for their completion. The median usability satisfaction in the SUS questionnaire was 55 (IQR 45.0-62.5).

A total of 150 tasks were carried out among all users and 89.3% (134/150) of the tasks were completed. Tasks 1 and 2 were completed 100% (60/60) of the time. Task 4 was completed at a lower proportion than the other tasks (22/30, 73.3%) and presented the highest number of errors (mean 2.5, SD 0.47). Task 3 and task 4 required personalized help to be completed (10/30, 33.3% and 8/30, 26.7%, respectively). The results of the different usability components are shown in Table 2 (also see Multimedia Appendix 4) and the details according to the different tasks are shown in Table 3 (also see Multimedia Appendix 5). The results of the subgroup usability analysis considering patients with cognitive deficits are provided in Multimedia Appendix 6.

In the multivariate analysis of the characteristics of the total tasks evaluated that were completed (Table 4), education level emerged as a facilitating factor to complete the task in an independent manner. Being male, having diseases related to cognition, and age hindered the completion of the task without help, with the latter factor being statistically significant.

Discussion

Main Findings

The usability effectiveness of the TeNDER technological tool was 89.3% (134/150). Overall, 40% (12/30) of the patients completed all tasks independently. Task 4 was completed at a lower proportion than the rest of the tasks (22/30, 73.3%) and presented the highest number of errors (mean 2.5, SD 0.47). The usability efficiency, evaluated as the median time to complete the total tasks, was 296.0 seconds (IQR 210.0-397.0), with an average value per task of 65.4 seconds (SD 92.7). The satisfaction in usability perceived by the patients was acceptable (mean 52.2, SD 16.9). Being male, having diseases related to cognition, and age were factors that hindered the completion of the task without personalized help, among which only age was statistically significant.

Comparison With Other Studies

The usability effectiveness of the TeNDER technological tool was 89.3%, which is similar to the results of previous studies carried out on different categories of patients for similar technological developments. Sánchez-Morillo et al [30] evaluated the usability of a technological tool aimed at monitoring the symptoms of patients with chronic obstructive pulmonary disease, and Georgsson et al [19] evaluated a system designed for the management of self-care in patients with diabetes. The proportion of tasks completed was 88% and 91%, respectively, despite the opposite characteristics of the participants with respect to the level of education and affinity for technology in each of the studies. As in our study, the degree of task complexity could have been adapted to the characteristics of the potential users: tasks 1 and 2 were completed by all patients, whereas the rest of the tasks, of greater complexity, were completed by only those with higher education. For those who did not have higher education, the task completion rate reached up to 55.6%. These differences in the use of technology depending on the level of education have been confirmed in previous studies [37].

Despite the high proportion of completed tasks, 56.7% of the patients required personalized help to complete at least one of the tasks. Older age and cognition-related diseases were risk factors for requiring personalized help to complete the tasks. Previous experience in evaluating the usability of a computerized system for self-care management aimed at patients with chronic diseases yielded similar percentages of effectiveness in usability and help for task completion [25], which points to the importance of having family members or professionals assist patients with chronic diseases to interact with a mobile app [38].

The median value for usability satisfaction was 55.0 (IQR 45.0-62.5), which is a low marginal score over not acceptable [32]. Ligons et al [21] obtained similar results and indicated that the degree of response in satisfaction with a technology or system may not be related to the ability for the completion of its tasks. That is, patients may be able to complete tasks without knowing why they have completed them or how they can benefit from them in their day-to-day lives. Other studies, including that of Sánchez-Morillo et al [30], suggested that high levels of satisfaction may be caused by the presence of qualified professionals who assisted during the usability evaluation.

The median age of the patients in our study was 75 years and a high degree of multimorbidity was notable, with a median of up to eight chronic diseases. Previous studies have analyzed the usability of a technology from the perspective of patients with a chronic index disease in particular [19,20,30,39]. For example, Wildenbos et al [22] analyzed how chronic diseases can affect the usability of technological tools. Thus, a single chronic disease can be the cause of physical, cognitive, and perception barriers [22]. Medical conditions that could favor the appearance of these barriers are represented in our study: diabetes, cardiovascular disease, cognitive impairment, and vision problems. However, as in previous studies, no differences were found in the different aspects of usability based on the number of chronic diseases.

Only 38 patients out of a total of 250 who signed prior informed consent agreed to participate in this usability study. It should be noted that a mobile instant messaging app was used as the method of offering participation and there was a nonresponse rate of 80% (200/250). This means of communication, although common in current society, could have caused a lack of confidence or security in patients [40]. Among the 38 patients who agreed to participate, 6 (15.2%) decided to leave the study as a result of the stress generated by the proposed tasks or due to lack of interest. Few previous usability studies have reported the number of losses [41], perhaps due to the small number of patients involved. Wildenbos et al [22] mentioned lack of motivation as a key element to achieving acceptance of technology by older people. The benefits of using a technology should be made evident quickly and easily; otherwise, feelings of frustration and of giving up its use are likely. In a time-limited usability evaluation, these benefits are not evident, and their nonparticipation can help to avoid feelings of uncertainty, wasting the time invested in learning a technology, or the shame of making mistakes.

Differences based on sex in the use of ICTs have been described in previous studies [37,42]. Among older people, access to technology and their degree of involvement in daily activity is greater in men than in women [37]. These differences are also identified in the different aspects of usability. In our study, the average number of errors committed per task was significantly higher in women (mean 1.4, SD 2.0) than in men (mean 0.6, SD 1.2). In addition, the proportion of women who needed personalized help to complete tasks was higher (10/15, 67%) than that in men (7/15, 47%). These differences have been largely justified by the fact that the labor participation of women has been lower, particularly in computerized jobs due to less training over the years [43].

Limitations

Although the number of participants in our study is similar to that of other studies and the findings obtained provide valuable information, a larger sample size would provide a larger data set to conduct more sophisticated and detailed statistical analyses. Moreover, given the characteristics of the research, it has not been possible to collect opinions, sensations, and emotions in relation to the technological tool that the patients experienced during task execution. For this reason, including a qualitative methodology such as focus groups [44], think-aloud tasks [45], or a user-centered cognitive walkthrough [20] could provide essential information for understanding the decision-making of patients with multimorbidity when faced with a mobile app aimed at health.

Another limitation identified is the time of tool use being limited to 48 hours. Studies such as those of Tahsin et al [46] and Baek et al [47] showed how usability results can change at different times over longer intervals of use for up to 1 year.

Conclusions

Although usability effectiveness was high, the poor efficiency and usability satisfaction results suggest that there are other factors that may interfere with these results. Sex and education level can influence the degree of completion of tasks. It has not been possible to show that multimorbidity is a key factor in the usability results of a technological tool.