Usability of a Co-designed eHealth Prototype for Caregivers: Combination Study of Three Frameworks

Introduction

Background

eHealth is becoming increasingly important to support people in taking care of their own health and that of their loved ones. In 2019, more than 90% of Canadians had access to the internet and 50% reported having access to at least one web-based health service [1]. The COVID-19 pandemic has amplified its use. eHealth is notably one of the solutions that can support caregivers in the daily tasks required to care for an older person at home [2]. Guay et al [3] suggest that internet-based interventions can have positive effects on the psychological well-being of caregivers of older persons. Irani et al [4] reported that people with chronic diseases and their caregivers were satisfied with the use of the technology. However, some faced technical challenges, whereas others were concerned about the technology’s lack of a personalized approach. Moreover, caregivers of functionally dependent older persons are often older themselves. In 2012, in Québec (Canada), 41% of caregivers were aged over 55 years [5]. A digital divide related to age and education [6], which are both determining factors of internet use [7], still remains. Concerns can, therefore, be raised about the acceptability of eHealth solutions within this group of users, as many factors influence older people’s acceptance of technology, such as privacy implications and usability factors [8].

In response to this issue, there is a growing interest in the co-design approach [9]. In this approach, researchers, designers, and participants are cocreating with users, who are considered experts of their experience and play a large role in knowledge development, idea generation, and concept development [10]. Authors have reported that the participation of different actors in a co-design project allowed a better understanding of each other’s perspective and reality [9,11]. As co-designers, caregivers and older adults can share their concerns and expectations about the technology in a democratic process, which might increase the fit between their needs and the system developed. However, the co-design approach cannot guarantee the usability and user experience (UX) of the designed system.

Usability and UX Evaluation

Usability Definition

Usability is the “functional relationships between people and the products and systems they use” [12]. It is also defined as the “extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [13]. Users do not want a difficult or uncomfortable experience in their interaction with the system, but usability requires more than just the users’ desire for it [14]. To achieve usability of the system, we need to evaluate it and adjust the design to address the problems found. Usability assessment is a complex analysis to perform, as evaluation criteria will differ depending on the usability framework (or set of criteria) used.

The International Organization for Standardization Framework

The ISO (International Organization for Standardization) usability framework (ISO 9241-210) is a framework accepted worldwide [13] to assess usability in general. The criteria of this framework are all part of the ISO’s definition of usability: effectiveness, efficiency, satisfaction, and context of use. The specified users,specified goals, and context of use are a combination of the situated aspects of the interaction with the system: who are the users, what do they want to achieve with the system, and in which context (at home or at work, on their phone or on their computer, etc)? All these factors need to be considered when assessing the usability of a system. Effectiveness is the accuracy and completeness with which users achieve specified goals. Are users able to achieve the task? Efficiency refers to the resources used in relation to the results achieved. How long and how easy was it to accomplish the task? Satisfaction is defined as the “extent to which the user’s physical, cognitive and emotional responses that result from the use of a system, product or service meet the user’s needs and expectations” [13]. Did users appreciate their interaction with the system while performing the task?

This framework allows a general picture of usability but cannot provide specific insights into what is needed to achieve better results. How can we fix a problem related to effectiveness? The Nielsen heuristic framework (1995) provides more details on what the system should do to meet the ISO criteria.

The Nielsen 10 Usability Heuristics Framework

Heuristics describe an approach to problem solving whereby people will rely on a limited number of principles to reduce the complexity of a task by predicting values to simpler judgmental operations [15]. Heuristics are helpful in predicting the reaction of users interacting with a system. The Nielsen framework (1995) listed 10 heuristics to consider while assessing or trying to achieve usability:

1. Visibility of system status: “The design should always keep users informed about what is going on, through appropriate feedback within a reasonable amount of time.”
2. Match between the system and the real world: “The design should speak the user’s language. Use words, phrases, and concepts familiar to the user, rather than internal jargon. Follow real-world conventions, making information appear in a natural and logical order.”
3. User control and freedom: “Users often perform actions by mistake. They need a clearly marked ‘emergency exit’ to leave the unwanted action without having to go through an extended process.”
4. Consistency and standards: “Users should not have to wonder whether different words, situations, or actions mean the same thing. [Words, situations, and actions should] follow platform and industry conventions.”
5. Error prevention: “Good error messages are important, but the best designs carefully prevent problems from occurring in the first place. Either eliminate error-prone conditions or check for them, and present users with a confirmation option before they commit to the action.”
6. Recognition rather than recall: “Minimize the user’s memory load by making elements, actions, and options visible. The user should not have to remember information from one part of the interface to another. Information required to use the design (e.g. field labels or menu items) should be visible or easily retrievable when needed.”
7. Flexibility and efficiency of use: “Shortcuts, hidden from novice users, may speed up the interaction for the expert user, such that the design can cater to both inexperienced and experienced users. [The system should] allow users to tailor frequent actions.”
8. Esthetic and minimalist design: “Interfaces should not contain information which is irrelevant or rarely needed. Every extra unit of information in an interface competes with the relevant units of information and diminishes their relative visibility.”
9. Help users recognize, diagnose, and recover from errors: “Error messages should be expressed in plain language (no error codes), precisely indicate the problem, and constructively suggest a solution.”
10. Help and documentation: “It’s best if the system doesn’t need any additional explanation. However, it may be necessary to provide documentation to help users understand how to complete their tasks [16].”

Refer to the NNGroup website [16] for a detailed description, with examples of each heuristic. These heuristics are guidelines for achieving effectiveness, efficiency, and satisfaction. They focus on the task goals. However, users’ goals are not always task oriented (do-goals) [17]. Other goals, such as be-goals, will affect their experience with technology [18]. Users are not just users. They are also human beings with feelings. The UX shifts the focus from the product to feelings while users interact with the technology [19].

Garrett Framework: The Elements of UX

The elements of UX framework [20] proposes 5 dimensions to describe UX design: strategy, scope, structure, skeleton, and surface. Each dimension has distinctive elements (Textbox 1).

Each framework provides a different understanding of how a product can get closer to what users really want. These frameworks inform different yet complementary aspects of usability and UX.

Objective

The objective of this study is to assess the usability and UX of an early version co-designed prototype to support the help-seeking process of caregivers of functionally dependent older persons. Trying to gather as much information as possible on potential improvements, we want to explore the contribution of the 3 frameworks presented: the ISO [13], Nielsen usability heuristics [16], and Garrett elements of UX [20].

Methods

Context of the Study

On the basis of the potential value of the co-design approach in the eHealth domain, we first co-designed an eHealth prototype to support the help-seeking process of caregivers of functionally dependent older persons (Figure 1). The co-design phase of the study (phase 2) included 8 co-design sessions and 3 advisory committee meetings held in 11 of the 16 administrative regions of the province of Québec from May 2017 to June 2018.

‎View this figure

The research project protocol of the entire study can be found in the paper by Latulippe et al [21], and results on user needs, requirements, and overall process and design decisions are presented in 3 papers [22-24]. A total of 74 co-designers were recruited, including 30 caregivers, 26 community workers, and 18 health professionals. Advisory committee meetings were held in plenary and co-design sessions were held in both plenary and subgroup workshops and included different types of activities depending on the objectives of the session.

The eHealth prototype developed was a website with 2 main objectives: helping caregivers to find resources (with a search tool and a questionnaire to help identify the needs) and allowing service providers to offer their services. The prototype is currently hosted on a private server.

Explorative Usability and UX Assessment

We chose an exploratory study method to assess usability, as the prototype was in its first version [25,26]. Changes were made to phase 3 of the initial protocol to gather more appropriate knowledge about usability and UX, considering the state of the prototype [21]. These changes include the addition of users’ first impressions, the accuracy of the methods used to perform the think-aloud method, and the use of a semistructured interview rather than a standardized questionnaire.

Recruitment

Participant recruitment included recruiting for the 2 objectives of the website (offering and finding resources for caregivers). Two researchers (KL and MC) completed the recruitment and data collection. All participants were recruited from a single region of Québec for feasibility reasons. The first inclusion criterion was potential users of the website. We contacted service providers via telephone and email. Service providers helped to recruit caregivers within their organization. One inclusion criterion for service providers was to provide services to caregivers of functionally dependent older persons. One inclusion criterion for caregivers was to provide assistance on a regular basis (at least once a week) to a person aged 65 years or older. Participating in phase 2 (co-design of the tool) was not an inclusion criterion for recruitment in the usability study (phase 3), but it was also not an exclusion criterion. As the objective of this study was exploratory and we were testing with an early version of the prototype, we targeted 10 users, including service providers and caregivers. We wanted an equal representation of participants for each objective of the website. Faulkner [27] revealed that the average percentage of problem areas found in 100 trials of 5 users found 85% (38/45), ranging from 55% (25/45) to nearly 100% (45/45), whereas groups of 10 found 95% (43/45) of the problems.

Data Collection

Global Process

As the prototype was in the early stages of development, the database did not contain any resources except the test resource entered by researchers during the programming of the prototype. Thus, we completed data collection in 2 parts through individual user testing, including 3 steps each (Figure 2). For the first part, we evaluated usability and UX for the offering resources objective. Participants contributed to adding some resources in the database in which caregivers would eventually search. For the second part, we evaluated usability and UX for the finding resources objective with caregivers. We collected data in French, the main language used in Québec.

‎View this figure

We completed the 3 steps of each part in a single session of 30-45 minutes with each participant. All sessions were videotaped. We conducted tests at the workplace for service providers and at home (or in a community center) for caregivers. Moderators used either a portable PC with a webcam and the Open Broadcast Studio (The OBS Project) [28] or the participant’s own computer, a camera or an iPad, and an audio recorder. Open Broadcast Studio is a free open-source software that allows the recording of multiple sources of data simultaneously. The webcam captures the participants’ reaction. The integrated audio captor records participants’ verbalization, and the desktop or browser windows are captured as another source.

Step 1: Users’ First Impressions

We used the 5-second test (5ST) to gather the first impression of users. This involved a display of the home page for 5 seconds, followed by questions [29,30]. The 5ST technique was used to gather information about general eye-catching attributes of the home page and provided general first impressions. After the 5 second display, we asked participant, “Can you tell me what you remember seeing?” To collect more detailed information about the perception of the utility for each user subgroup, we added a second display of the home page without a time limit. Participants were then asked to express their perceived usefulness, “As a [caregiver or service provider], what do you think you could do with this website?”

Step 2: Task Scenarios

The second step included task scenarios [25] with a coaching think-aloud protocol [31]. The task scenarios represented several tasks that the user would typically perform with this website and put them in context [25]. Scenarios differed depending on the targeted subgroup. Each scenario included a practice task to familiarize participants with the think-aloud protocol, followed by 5 assessed tasks. We selected the tasks based on what each subgroup of users would typically want to do with the website and on specific interactions that the research team wanted to assess (Textbox 2). In accordance with the coaching think-aloud protocol, moderators worked with participants during task performance. When the participant stopped talking during the task, the moderator repeated the instruction, keep thinking aloud, please. When a participant struggled with a task, the moderator provided some guidance.

Step 3: Semistructured Interview

We created an interview guide based on validated usability questionnaires [32-35]. We created our own interview guide because validated questionnaires have limited applicability and are not suited to all systems [36]. We also wanted to address the specific objectives of the usability evaluation and UX of our prototype, such as problems faced during task performance [25]. We included 8 questions, with probing questions adapted to our designed prototype in the interview guide to answer more specifically to our study objectives (Textbox 3).

To collect sociodemographic data, participants were asked questions about their age, profession, education level, and profession. We also asked 3 multiple-choice questions to assess participants’ perceptions of their technology profiles (Textbox 4).

Data Analysis

One researcher (corresponding author) performed the data analysis. We conducted qualitative data analysis in Microsoft Excel using the video recordings of each session. For the first (user impression) and third (semistructured interviews) steps, we conducted an inductive thematic analysis [37]. All participant verbalizations (answers) were transcribed, and some answers were translated by the author for publication purposes. We numbered each answer and collected a list of 5 data items for each (Textbox 5). We used the filter functionality to group and analyze the data.

For the second step (task scenarios), we conducted a deductive analysis [37] based on each criterion of the 3 frameworks [13,16,20]. We entered participant observations and verbalizations for each task. We numbered each problem found during the tasks and registered details for 12 items, each being a column in the Microsoft Excel spreadsheet. Textbox 5 presents a list of the items collected for each step.

For step 2, we first coded each problem according to one criterion of each framework. We then combined the coding for all frameworks using the pivot table functionality. We selected the Garrett [20] criteria to organize the identified heuristic [16] of each problem in the rows field and crossed them with ISO categories [13] in the columns field in the Microsoft Excel spreadsheet.

The study received ethical approval from the Comité d’éthique de la recherche sectoriel santé des populations et première ligne (2016-2017-10 MP). Informed consent was obtained from each participant, who also received a nominal compensation of Can $20 (US $16.45).

Results

Participants’ Demographics

We recruited a total of 10 participants: 4 caregivers and 6 service providers. We conducted 5 user tests for each phase (part 1: offering resources and part 2: finding resources; Table 1). One service provider participated as if she were a caregiver (phase 2). She represented what could happen in a real-context setting: a service provider helping a caregiver.

Step 1: Users’ First Impressions

Table 2 presents the results (emerging categories) for the 5ST of the home page. Elements most commonly identified by participants were located in the top-right section of the page (n=10).

The top left section, where the search tool is located, was among the elements that were perceived less frequently. It was perceived by 7 participants, with only 4 participants mentioning the search tool, which corresponds to the finding resources objective. Figure 3 shows the home page displayed to the participants.

‎View this figure

After the 5ST, for the second home page display (without a time limit), all service providers answered that they thought the website was for finding resources to refer or help caregivers. One service provider also mentioned the requirement of registering its organization. All caregivers answered that they could search or find resources or service providers; 2 of them insisted on the fact that they could find respite resources.

Step 2: Task Scenarios

Summary

We identified 151 problems with the task scenarios, which were classified according to the categories of the 3 frameworks. Some participants were not able to perform all the tasks because of programming errors. When problems were programming errors, they were classified as N/A (not applicable) in each category. This was also done if the focus on the task seemed to interfere with the participant’s perception, for example, if the participant did not see an item they were asked about (the phone number of the helpline for caregivers) because the focus was not on that item but on the task (finding a resource in their region). In those cases, we did not classify the problems, as the task could have affected the interpretation of a category. Most problems (49/151, 32.5%) occurred during the creation of a profile for service providers. The other tasks with a considerable number of problems for service providers were finding resources (18/151, 11.9%) and adding an activity (10/151, 6.6%). For caregivers, 11.3% (17/151) of problems occurred during the task finding a resource in their region and 8.6% (13/151) of problems during the task identifying their own needs. The following sections provide a description of these problems.

ISO Classification

Table 3 presents findings related to the ISO usability framework. The table presents the number of problems in each category, an example for each with data input number, participant ID, and the transcript or description of the observation. Service providers are identified as SP# and caregivers as C#. It also presents the task where the problems occurred and the type of problem.

Almost half of the problems (68/151, 45%) were classified as effectiveness and occurred mainly for service providers during part 1 (50/68, 74%). A total of 32.5% (49/151) were classified as efficiency, again mostly by service providers (31/49, 63%). A few problems (18/151, 11.9%) were related to satisfaction. Four problems were classified as context of use. These problems were identified by a single participant, and most of them (3/4, 75%) could be explained by the fact that this participant was visually impaired. The problems coded as context of use included the following: (1) the participant tried to connect or create a profile using the caregiver access instead of the service provider access, (2) insufficient color contrast, (3) and the size of user interface elements.

Nielsen Heuristic Classification

Regarding the classification using Nielsen heuristics, match between the system and the real world was the most important heuristic principle identified, representing 19.9% (30/151) of problems (Table 4). This heuristic was mostly identified for problems arising for service providers (21/30, 70%) when performing the create a profile task (12/30, 40%). Other frequently mentioned usability principles included help and documentation (25/151, 16.6%), user control and freedom (22/151, 14.6%), error prevention (21/151, 13.9%), and recognition rather than recall (18/151, 11.9%), all of which were again mostly for service providers during the create a profile task.

Garrett Elements of UX Classification

Table 5 presents the results of the analysis using the Garrett framework. Most of the problems (113/151, 75.8%) were classified in the skeleton plane, especially in the interface design (54/151, 35.8%) and information design (50/151, 33.1%) dimensions. For interface design, 65% (35/54) of problems occurred for service providers, mostly to create a profile (20/54, 37%). For information design, 80% (40/50) of problems occurred for service providers, mostly to create a profile (24/50, 48%). Only one problem was categorized in the user needs category, and it was also classified for context of use in the ISO categorization. This problem reflects an accessibility problem for visually impaired users, meaning that the prototype did not address the specific needs of visually impaired users.

Combining the Frameworks

Combining all 3 frameworks of analysis provides a comprehensive picture of the identified problems. Multimedia Appendix 1 presents the problems in terms of the dimension of the Garrett framework and the category of ISO usability criteria. The combination also identifies the Nielsen heuristic usability guideline the problem does not address. This table indicates that for interface design, problems were mainly identified for effectiveness (22/54, 41%) and efficiency (24/54, 44%), with several problems of error prevention (ie, when trying to connect to their profile). Problems are also found for effectiveness (19/50, 38%) and efficiency (25/50, 50%) in information design, concerning especially the match between the systemand the real world, mostly for service providers during the creation of their profile and the help and documentation (eg, caregiver was looking for the word Respite and did not think of entering it in the search engine). The combination of frameworks allows a better understanding of usability problems and provides greater insight into the improvements needed. The numbers in the cells indicate the number of problems at the intersection of the row and the column.

Step 3: Semistructured Interview

Overview

This section presents the questions and translations of the transcripts, including answers to each question. Service providers are identified as SP [#] and caregivers as C [#]. Question 1 was asked during task performance, and the results were included at that point.

Question 2

The second question was as follows: “Is the website easy to use?”—Seven participants answered this question. Answers varied among participants: 3 answered “yes” (SP4, C1, and C5), 1 specifying that it was easy to understand and that the screen was not overloaded (C1). Two answers seemed ambivalent:

We find resources when it [website] works properly. If I am looking for an organization but I don’t know the organizations...Finding resources, I don’t know the resources, it is not clear.
[SP1]

Middle. Knowing if it [the search] worked or not. But visually it’s quite easy, not overloaded. It is easy to search.
[C3]

One participant answered negatively:

It makes me feel incompetent. I can’t immediately find what I’m looking for.
[SP5]

Question 3

The third question was: “Is the organization of the website logical and optimal?”—Seven participants answered this question. Two were positive:

Personally, I think it’s OK. I will sit with the caregiver and find resources.
[C5]

Oh yes! We have a lot to learn. If I had it, I would learn a lot!
[C4]

Three participants said it was clear, but not optimal (SP4, SP5, and C3). Similarly, 2 participants mentioned it could be better (SP2), 1 commenting on the information: “For me, information needs to be precise, I don’t want to get lost in things that will take time” [C2].

Question 4

Next, the following question was asked to six participants: “Did you find information easily?”—Only 1 answer was fully positive: “Yes, indeed. It should have info for each organization” [C1].

Other answers were more mitigated: “Yes and no. For now, there is not a lot in it” [SP4], with some commenting on the fact that they needed the interviewer to complete the task (C3 and C4). Two were negative:

Not really. I was not able to get results.
[SP2]

It was too long.
[C2]

One participant commented on the information he was not able to find easily:

I expected to arrive directly in the activities, because now, I have to go through the list before getting to the activities. Especially because it is presented...In the list, the organizations were first, then the activities and after the documents. There should be some logic to it.
[C1]

Question 5

The fifth question was as follows: “When navigating on the website, is it easy to know where you are?”—All participants who answered this question (n=8) said they were mostly able to figure out where they were. Three participants answered “yes” (SP2, SP4, and C5); one answered “Quite easily” [C2]; and another answered “Yes, I think” [C4]. Other answers were ambivalent, with participants providing some explanation:

The first time no, but after yes.
[C3]

It’s because we could not see the rest. I knew where I was in the section I could reach.
[SP1]

I knew where I was in the website. But, when I clicked here [browser Back button], I expected to go back to the page I was before, but it brings me back to the beginning.
[C1]

Question 6

The following question was answered by only 1 participant: “Generally speaking, are you satisfied with this website?”

More or less. Contrasts should be adjusted for older and colour-blind people.
[SP4]

Question 7

The following question was: “Do you feel comfortable using this website?”—Two of those who answered this question (n=4) answered “yes” (SP4 and C3). One specified that she would use it with a digital tablet (C2). Another (SP2) mentioned that he would be somewhat comfortable using it, even with the current problems. He was referring to one of the programming problems.

Question 8

For the last question: “Would you like to use this website for your tasks?”—Again, 2 (of a total of 3) participants answered simply “yes” (SP4 and C3). One participant indicated that he would use it with caregivers:

Of course, I would use it! I would use it with the caregiver to help him develop his ability to find information with this tool.
[SP6]

Discussion

Principal Findings

This study aims to assess the usability and UX of the 2 objectives of an early co-designed prototype. Findings from step 1 (users’ first impressions) indicate that participants were able to identify the 2 main objectives of the website. Moreover, even if participants were ambivalent regarding information retrieval, (answers from question 3 during step 3) and the ease of use of the website (answers for question 2 during step 3), they were still comfortable and interested in using the website (answers from questions 5 to 7 during step 3). On the other hand, results from the task scenarios (step 2) tend to indicate that there were more usability problems for the offering resources objective, especially when service providers were trying to create their profile. However, this was not the website’s main objective, as the co-design study first aimed to conceive an eHealth tool to support the help-seeking process of caregivers. The second objective (offering resources) emerged early during the co-design process, that is, during the identification of functional and content requirements [23]. As mentioned by Luck [38], in participatory design research, knowledge is constructed through practice, and one cannot entirely foresee the direction of the experiment. This was the case, for example, for the co-design study by Tironi [39], in which new knowledge about the ontological perspective of users forced the redefinition of the initial protocol.

Required Improvements on Accessibility

A second finding relates to accessibility. Accessibility “means removing barriers that might prevent someone from using something, regardless of their access needs” [40]. Accessibility problems were found during task scenarios (step 2) for 1 service provider. Due to a visual impairment, this participant was unable to see the entire page at once. The participant was using a special device to enlarge the interface on the screen. Even if it is uncommon to identify context of use as a usability category (other studies generally use the effectiveness, efficiency, satisfaction triad), we chose to include it in our study to see whether we would be able to classify problems in that category, and we were able to do so. The special needs of this participant were not addressed. We recognize that no participant with a visual impairment was included during the co-design process [22]. To maximize the potential of addressing all user needs, participants with special needs should have been included in the co-design process. As mentioned by Cahill [41], co-design or participatory action research is precisely an appropriate method for including excluded perspectives and challenging typical knowledge production. From a social justice perspective, other co-design studies should include users with impairments, as special techniques to co-design with them are offered in the literature [42,43].

Combination of Frameworks

The combination of the 3 frameworks was a novel contribution. It has broadened the perspective and enhanced the strength of our study. As pointed out by Lacerda and von Wangenheim [44] in their systematic literature review, current usability models have many problems (lack of information on the intended use, data collection instruments, and assessment process), leading researchers to seek other sources or combine different models and methods. In our study, the use of the ISO framework was particularly helpful in revealing an important accessibility issue. The use of the Garrett framework was decisive in identifying the dimensions in which the problem was located. Nielsen heuristics helped us to understand how to improve the website in further iterations of the prototype. Each framework provided useful insights to understand the usability and UX of our prototype. However, the combination of the analyses of all 3 frameworks was even more informative. For example, we were able to identify that most problems were located in the interface and information design and were effectiveness problems (users being unable to complete the task) or efficiency problems (the level of difficulty in performing the task). Moreover, we were able to get a better idea of how to address the problems, knowing which heuristics they were not addressing, which were often the match between the system and the real world and help and documentation.

Challenges and Limitations

This study has some limitations. First, scenarios were created by the research team and imposed on the participants. This may have affected the results, as the focus was on the task and might have hindered access to other useful information. Second, the data were analyzed by only 1 expert or researcher. The results were presented to the research team, who agreed on the big picture without determining proper intercoder agreement. Third, programming errors interfered with some tasks, which may have led to the loss of useful information on usability. These problems were identified as programming errors, but the participants were not able to perform the task. If it had been possible to perform the task, other usability problems might have been identified for the task. During the analysis, we also realized that there was a possible mapping issue between the different categories of the frameworks. For example, “The participant did not see the phone number” could have been identified as either effectiveness or efficiency. The interpretation relies on what the analyst was focusing on. It could be coded as effectiveness if we consider that the user needs to call the phone number, and it could be coded as efficiency if the phone number is one method (among several) to access information. Thus, the combination of the 3 conceptual frameworks does not bring a mutually exclusive categorization, but it reduces the risk of blind spots. Regardless of the category, we were still able to identify that the phone number was not perceived and needed more emphasis.

Other limitations were related to the study participants. First, the participants had a high level of education, as most had a university degree (n=8). Although this might represent the population of service providers, it is not representative of the caregiver population. In Québec, only 27.6% of caregivers had a university degree [45]. The second limitation was the age gap between the offering resources group and the finding resources group. Age is a determining factor in the use of internet products [7]. However, the finding resources group was mostly caregivers of functionally dependent older persons. Statistics indicate that these caregivers are often older themselves [5], which could explain the gap. The third limitation was the number of participants. Although we had 10 users, all of them were not testing exactly the same pages. Five users were testing each objective of the website. Nevertheless, looking at the results, we still consider that most of the problems seemed to emerge during this usability and UX assessment, without an absolute confirmation on the saturation of problems. As this study was exploratory and targeted an early version of the website, we are confident that we have collected sufficient information to improve the prototype.

Our results are transferable to a very limited extent to other eHealth systems. They are indeed related to a specific interactive product (website) dedicated to specific users (caregivers and service providers). Nevertheless, our analysis proposition combining the ISO [13], Nielsen heuristic [16], and elements of UX [20] is highly applicable to the usability or UX evaluation of other eHealth systems.

Conclusions

This study provided improvement possibilities for a prototype co-designed with caregivers and service providers. We were able to identify several usability and UX problems. The 3 frameworks used for the analysis allowed us to understand the nature of the problem (ISO) [13] and the dimension where it lies (elements of UX) [20], as well as provide potential problem-solving solutions based on the predicted judgmental operations (Nielsen heuristics) [16]. Thus, we will continue the co-design process to address those problems by recruiting service providers and caregivers to co-design a new version of the prototype. Our analytical method, based on the 3 conceptual frameworks and their combination, broadened the perspective of the problems encountered. This combination of frameworks for usability and UX analysis is a novel contribution that is transferable to other eHealth systems, which contributes to the advancement of knowledge in the eHealth community.