eHealth is becoming increasingly important to support people in taking care of their own health and that of their loved ones. In 2019, more than 90% of Canadians had access to the internet and 50% reported having access to at least one web-based health service [1]. The COVID-19 pandemic has amplified its use. eHealth is notably one of the solutions that can support caregivers in the daily tasks required to care for an older person at home [2]. Guay et al [3] suggest that internet-based interventions can have positive effects on the psychological well-being of caregivers of older persons. Irani et al [4] reported that people with chronic diseases and their caregivers were satisfied with the use of the technology. However, some faced technical challenges, whereas others were concerned about the technology’s lack of a personalized approach. Moreover, caregivers of functionally dependent older persons are often older themselves. In 2012, in Québec (Canada), 41% of caregivers were aged over 55 years [5]. A digital divide related to age and education [6], which are both determining factors of internet use [7], still remains. Concerns can, therefore, be raised about the acceptability of eHealth solutions within this group of users, as many factors influence older people’s acceptance of technology, such as privacy implications and usability factors [8].

In response to this issue, there is a growing interest in the co-design approach [9]. In this approach, researchers, designers, and participants are cocreating with users, who are considered experts of their experience and play a large role in knowledge development, idea generation, and concept development [10]. Authors have reported that the participation of different actors in a co-design project allowed a better understanding of each other’s perspective and reality [9,11]. As co-designers, caregivers and older adults can share their concerns and expectations about the technology in a democratic process, which might increase the fit between their needs and the system developed. However, the co-design approach cannot guarantee the usability and user experience (UX) of the designed system.

The objective of this study is to assess the usability and UX of an early version co-designed prototype to support the help-seeking process of caregivers of functionally dependent older persons. Trying to gather as much information as possible on potential improvements, we want to explore the contribution of the 3 frameworks presented: the ISO [13], Nielsen usability heuristics [16], and Garrett elements of UX [20].

On the basis of the potential value of the co-design approach in the eHealth domain, we first co-designed an eHealth prototype to support the help-seeking process of caregivers of functionally dependent older persons (Figure 1). The co-design phase of the study (phase 2) included 8 co-design sessions and 3 advisory committee meetings held in 11 of the 16 administrative regions of the province of Québec from May 2017 to June 2018.

The research project protocol of the entire study can be found in the paper by Latulippe et al [21], and results on user needs, requirements, and overall process and design decisions are presented in 3 papers [22-24]. A total of 74 co-designers were recruited, including 30 caregivers, 26 community workers, and 18 health professionals. Advisory committee meetings were held in plenary and co-design sessions were held in both plenary and subgroup workshops and included different types of activities depending on the objectives of the session.

The eHealth prototype developed was a website with 2 main objectives: helping caregivers to find resources (with a search tool and a questionnaire to help identify the needs) and allowing service providers to offer their services. The prototype is currently hosted on a private server.

We chose an exploratory study method to assess usability, as the prototype was in its first version [25,26]. Changes were made to phase 3 of the initial protocol to gather more appropriate knowledge about usability and UX, considering the state of the prototype [21]. These changes include the addition of users’ first impressions, the accuracy of the methods used to perform the think-aloud method, and the use of a semistructured interview rather than a standardized questionnaire.

Participant recruitment included recruiting for the 2 objectives of the website (offering and finding resources for caregivers). Two researchers (KL and MC) completed the recruitment and data collection. All participants were recruited from a single region of Québec for feasibility reasons. The first inclusion criterion was potential users of the website. We contacted service providers via telephone and email. Service providers helped to recruit caregivers within their organization. One inclusion criterion for service providers was to provide services to caregivers of functionally dependent older persons. One inclusion criterion for caregivers was to provide assistance on a regular basis (at least once a week) to a person aged 65 years or older. Participating in phase 2 (co-design of the tool) was not an inclusion criterion for recruitment in the usability study (phase 3), but it was also not an exclusion criterion. As the objective of this study was exploratory and we were testing with an early version of the prototype, we targeted 10 users, including service providers and caregivers. We wanted an equal representation of participants for each objective of the website. Faulkner [27] revealed that the average percentage of problem areas found in 100 trials of 5 users found 85% (38/45), ranging from 55% (25/45) to nearly 100% (45/45), whereas groups of 10 found 95% (43/45) of the problems.

One researcher (corresponding author) performed the data analysis. We conducted qualitative data analysis in Microsoft Excel using the video recordings of each session. For the first (user impression) and third (semistructured interviews) steps, we conducted an inductive thematic analysis [37]. All participant verbalizations (answers) were transcribed, and some answers were translated by the author for publication purposes. We numbered each answer and collected a list of 5 data items for each (Textbox 5). We used the filter functionality to group and analyze the data.

For the second step (task scenarios), we conducted a deductive analysis [37] based on each criterion of the 3 frameworks [13,16,20]. We entered participant observations and verbalizations for each task. We numbered each problem found during the tasks and registered details for 12 items, each being a column in the Microsoft Excel spreadsheet. Textbox 5 presents a list of the items collected for each step.

For step 2, we first coded each problem according to one criterion of each framework. We then combined the coding for all frameworks using the pivot table functionality. We selected the Garrett [20] criteria to organize the identified heuristic [16] of each problem in the rows field and crossed them with ISO categories [13] in the columns field in the Microsoft Excel spreadsheet.

The study received ethical approval from the Comité d’éthique de la recherche sectoriel santé des populations et première ligne (2016-2017-10 MP). Informed consent was obtained from each participant, who also received a nominal compensation of Can $20 (US $16.45).

We recruited a total of 10 participants: 4 caregivers and 6 service providers. We conducted 5 user tests for each phase (part 1: offering resources and part 2: finding resources; Table 1). One service provider participated as if she were a caregiver (phase 2). She represented what could happen in a real-context setting: a service provider helping a caregiver.

Table 2 presents the results (emerging categories) for the 5ST of the home page. Elements most commonly identified by participants were located in the top-right section of the page (n=10).

The top left section, where the search tool is located, was among the elements that were perceived less frequently. It was perceived by 7 participants, with only 4 participants mentioning the search tool, which corresponds to the finding resources objective. Figure 3 shows the home page displayed to the participants.

After the 5ST, for the second home page display (without a time limit), all service providers answered that they thought the website was for finding resources to refer or help caregivers. One service provider also mentioned the requirement of registering its organization. All caregivers answered that they could search or find resources or service providers; 2 of them insisted on the fact that they could find respite resources.

This study aims to assess the usability and UX of the 2 objectives of an early co-designed prototype. Findings from step 1 (users’ first impressions) indicate that participants were able to identify the 2 main objectives of the website. Moreover, even if participants were ambivalent regarding information retrieval, (answers from question 3 during step 3) and the ease of use of the website (answers for question 2 during step 3), they were still comfortable and interested in using the website (answers from questions 5 to 7 during step 3). On the other hand, results from the task scenarios (step 2) tend to indicate that there were more usability problems for the offering resources objective, especially when service providers were trying to create their profile. However, this was not the website’s main objective, as the co-design study first aimed to conceive an eHealth tool to support the help-seeking process of caregivers. The second objective (offering resources) emerged early during the co-design process, that is, during the identification of functional and content requirements [23]. As mentioned by Luck [38], in participatory design research, knowledge is constructed through practice, and one cannot entirely foresee the direction of the experiment. This was the case, for example, for the co-design study by Tironi [39], in which new knowledge about the ontological perspective of users forced the redefinition of the initial protocol.

A second finding relates to accessibility. Accessibility “means removing barriers that might prevent someone from using something, regardless of their access needs” [40]. Accessibility problems were found during task scenarios (step 2) for 1 service provider. Due to a visual impairment, this participant was unable to see the entire page at once. The participant was using a special device to enlarge the interface on the screen. Even if it is uncommon to identify context of use as a usability category (other studies generally use the effectiveness, efficiency, satisfaction triad), we chose to include it in our study to see whether we would be able to classify problems in that category, and we were able to do so. The special needs of this participant were not addressed. We recognize that no participant with a visual impairment was included during the co-design process [22]. To maximize the potential of addressing all user needs, participants with special needs should have been included in the co-design process. As mentioned by Cahill [41], co-design or participatory action research is precisely an appropriate method for including excluded perspectives and challenging typical knowledge production. From a social justice perspective, other co-design studies should include users with impairments, as special techniques to co-design with them are offered in the literature [42,43].

The combination of the 3 frameworks was a novel contribution. It has broadened the perspective and enhanced the strength of our study. As pointed out by Lacerda and von Wangenheim [44] in their systematic literature review, current usability models have many problems (lack of information on the intended use, data collection instruments, and assessment process), leading researchers to seek other sources or combine different models and methods. In our study, the use of the ISO framework was particularly helpful in revealing an important accessibility issue. The use of the Garrett framework was decisive in identifying the dimensions in which the problem was located. Nielsen heuristics helped us to understand how to improve the website in further iterations of the prototype. Each framework provided useful insights to understand the usability and UX of our prototype. However, the combination of the analyses of all 3 frameworks was even more informative. For example, we were able to identify that most problems were located in the interface and information design and were effectiveness problems (users being unable to complete the task) or efficiency problems (the level of difficulty in performing the task). Moreover, we were able to get a better idea of how to address the problems, knowing which heuristics they were not addressing, which were often the match between the system and the real world and help and documentation.

This study has some limitations. First, scenarios were created by the research team and imposed on the participants. This may have affected the results, as the focus was on the task and might have hindered access to other useful information. Second, the data were analyzed by only 1 expert or researcher. The results were presented to the research team, who agreed on the big picture without determining proper intercoder agreement. Third, programming errors interfered with some tasks, which may have led to the loss of useful information on usability. These problems were identified as programming errors, but the participants were not able to perform the task. If it had been possible to perform the task, other usability problems might have been identified for the task. During the analysis, we also realized that there was a possible mapping issue between the different categories of the frameworks. For example, “The participant did not see the phone number” could have been identified as either effectiveness or efficiency. The interpretation relies on what the analyst was focusing on. It could be coded as effectiveness if we consider that the user needs to call the phone number, and it could be coded as efficiency if the phone number is one method (among several) to access information. Thus, the combination of the 3 conceptual frameworks does not bring a mutually exclusive categorization, but it reduces the risk of blind spots. Regardless of the category, we were still able to identify that the phone number was not perceived and needed more emphasis.

Other limitations were related to the study participants. First, the participants had a high level of education, as most had a university degree (n=8). Although this might represent the population of service providers, it is not representative of the caregiver population. In Québec, only 27.6% of caregivers had a university degree [45]. The second limitation was the age gap between the offering resources group and the finding resources group. Age is a determining factor in the use of internet products [7]. However, the finding resources group was mostly caregivers of functionally dependent older persons. Statistics indicate that these caregivers are often older themselves [5], which could explain the gap. The third limitation was the number of participants. Although we had 10 users, all of them were not testing exactly the same pages. Five users were testing each objective of the website. Nevertheless, looking at the results, we still consider that most of the problems seemed to emerge during this usability and UX assessment, without an absolute confirmation on the saturation of problems. As this study was exploratory and targeted an early version of the website, we are confident that we have collected sufficient information to improve the prototype.

Our results are transferable to a very limited extent to other eHealth systems. They are indeed related to a specific interactive product (website) dedicated to specific users (caregivers and service providers). Nevertheless, our analysis proposition combining the ISO [13], Nielsen heuristic [16], and elements of UX [20] is highly applicable to the usability or UX evaluation of other eHealth systems.

This study provided improvement possibilities for a prototype co-designed with caregivers and service providers. We were able to identify several usability and UX problems. The 3 frameworks used for the analysis allowed us to understand the nature of the problem (ISO) [13] and the dimension where it lies (elements of UX) [20], as well as provide potential problem-solving solutions based on the predicted judgmental operations (Nielsen heuristics) [16]. Thus, we will continue the co-design process to address those problems by recruiting service providers and caregivers to co-design a new version of the prototype. Our analytical method, based on the 3 conceptual frameworks and their combination, broadened the perspective of the problems encountered. This combination of frameworks for usability and UX analysis is a novel contribution that is transferable to other eHealth systems, which contributes to the advancement of knowledge in the eHealth community.