Published on in Vol 10 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/37785, first published .
Involving Older People With Frailty or Impairment in the Design Process of Digital Health Technologies to Enable Aging in Place: Scoping Review

Involving Older People With Frailty or Impairment in the Design Process of Digital Health Technologies to Enable Aging in Place: Scoping Review

Involving Older People With Frailty or Impairment in the Design Process of Digital Health Technologies to Enable Aging in Place: Scoping Review

Review

1Department of Public Health, University of Copenhagen, Copenhagen, Denmark

2Department of Health, SINTEF Digital, Oslo, Norway

3School of Nursing, McMaster University, Hamilton, ON, Canada

4Smart Homes, Eindhoven, Netherlands

Corresponding Author:

Emilie Kauffeldt Wegener, MA, MSc

Department of Public Health

University of Copenhagen

Øster Farimagsgade 5

Copenhagen, 1353

Denmark

Phone: 45 26202168

Email: emilie.wegener@sund.ku.dk


Background: With an increase in life expectancy globally, the focus on digital health technologies that can enhance physical and mental health among older people with frailty and impairment has increased. Similarly, research interest in how digital health technology can promote well-being and self-management of health in older age has increased, including an increased focus on methods for designing digital health technologies that meet the various medical, psychological, and social needs of older population. Despite the increased focus, there remains a necessity to further understand the needs of this population group to ensure uptake and to avoid introduction of additional challenges when introducing technologies, for example, because of poor technological design. The scope is limited to digital health technologies meant to enable older people with frailty and impairment to age in place.

Objective: In this study, we aimed to explore how older people with frailty and impairment are involved in various parts of the design processes of digital health technologies and identify gaps or neglected steps in a user-involving design process. This included a focus on recruitment strategies, contributions, and methods used to address the perspectives, needs, and desires of older people with frailty and impairment in the development of digital health technologies.

Methods: A scoping review was conducted in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) reporting from February 2021 to April 2021. Literature searches were conducted in PubMed, Scopus, Embase, and IEEE using a search string covering the concepts of health technology, older people, frailty and impairment, user-centered design, and self-management.

Results: In total, 1891 studies were imported for screening from the initial search. A total of 22 studies were included in this review after full-text screening and manual search. Invitation through partners was the most reported recruitment strategy to involve older people with frailty and impairment in the design process of digital health technologies. Furthermore, they were commonly involved in the final evaluation of the development process. Three main gaps identified were the use of outreach approaches to recruit older people with frailty and impairment in the design process of digital health technologies, description of the value of involvement and outcome of the contribution of participants, and knowledge regarding involvement in all parts of the design process.

Conclusions: Although there is literature on methods for involving older people with frailty and impairment in the design of digital health technology, there is little methodological dialogue on the nuances of how different methods for involvement relate to and shape the outcome of the development process.

JMIR Hum Factors 2023;10:e37785

doi:10.2196/37785

Keywords



Background

According to the data from the World Health Organization (WHO), the global population aged ≥60 years will increase from 12% to 22% between 2015 and 2050 [1]. This change is further challenged by the existing household structures in the European Union (EU) with increasing numbers of older people living alone. One approach to addressing the known challenges associated with this growing population is to support older people to age in place, which is defined as “the ability to live in one’s own home and community safely, independently, and comfortably, regardless of age, income, or ability level” [2]. This can be achieved by creating external environments that support social activities within local communities or by introducing ambient and assistive technologies to support everyday life and activities, often referred to as gerontechnology [3-6]. Of particular importance and specific to this review are technologies, often referred to as digital health technologies, that relate to the management of health conditions in older people [7].

Inappropriate Technological Design

The design of inappropriate technologies can limit uptake and enhance disability and inequity among older people with frailty and impairment [8]. For example, technologies meant to enhance safety and enable independence among older people with cognitive impairment can be disempowering or dehumanizing if designed and used inappropriately [8,9].

The introduction of new technologies that do not address or fully understand the needs of the end user may pose a challenge [10]. For older people, these challenges may be amplified owing to preexisting impairments or frailty. In using the term frailty, we refer to “a state of physiological vulnerability with diminished capacity to manage external stressors,” which can increase the risk of illnesses, falls, disability, and death [11]. The limited uptake of new technologies among older people has been associated with a misalignment in perceptions between those developing the technologies and older end users [12]. This misalignment frequently leads to either limited uptake or outright rejection [13-17]. Therefore, the involvement of older people who are frail or impaired (ie, experiencing physical or mental impairments such as dementia, aphasia, motor dysfunction, ataxia, hearing, or visual loss) in the development of technology is not only necessary in maximizing uptake but also in realizing the intent of technology to mitigate frailty and enable older people to manage challenges of everyday life despite impairments.

Involving Older People With Frailty and Impairment

The Food and Drug Administration has provided recommendations on patient engagement in the design and conduct of medical device clinical studies, including obtaining input from patients through meetings, home visits, or web-based follow-up and discussing barriers for recruitment with patient advisers [18]. In addition, the International Organization for Standardization provides guidance on how to ensure the design of products and services with the involvement of end users [19]. These guidelines and recommendations for involving end users in the development of health technology support this work and also demonstrate the increased global attention on this important work.

Although it has been widely accepted to involve end users in the design of new technologies and an increased focus on user involvement among certifying bodies is emerging [20,21], there are no standardized requirements or guidelines on how to involve end users [22]. Also, the academic debate on appropriate methods that involve older people who are physically or cognitively impaired or otherwise understood to be frail and allow them to express their needs and desires vis-à-vis technology is lagging. Consequently, there is a need for further knowledge about how to involve older people with frailty and impairment in the design of technology to ensure that their needs and desires are addressed and to better understand what they find meaningful to increase the likelihood of technology adoption.

Digital Health Technologies

In this scoping review, the focus is limited to how older people with frailty and impairment are involved in the design process of digital health technologies. Digital health technologies are defined by the Food and Drug Administration as “the use of computing platforms, connectivity, software, and sensors for health care and related uses” [7]. In this review, we include eHealth and its underlying terms in our understanding of digital health technologies. “eHealth” is defined by WHO as an umbrella term covering the general use of technologies for health care–related processes, including mobile health, the use of different mobile-based solutions, telemedicine, remote clinical services, and telehealth covering both remote and nonremote clinical services [23].

Aim

The identification and application of purposeful methods for involving older people with frailty and impairment in the innovation and implementation of digital health technology may be a promising means of ensuring that the technology can fulfill its purpose of enabling such older people to age in place with dignity and on their own terms.

Against this background, this review aimed to explore how older people with frailty and impairment are involved in various aspects of design processes of digital health technologies. This was done to identify gaps or neglected steps in a user-involving design process. This included a focus on recruitment strategies, contributions, and methods used to address the perspectives, as well as the needs and desires of older people with frailty and impairment in the development of digital health technologies.

To this end we pursued the following research questions:

  1. What are the characteristics of the participants included in the design processes and how are they recruited?
  2. Based on the objective of this study, what are the outcomes? What was the technology developed and how did the participants contribute?
  3. What kinds of methods and activities have been used to involve older people with frailty and impairment and when were they involved during the development process?

Overview

The scoping review was conducted as part of an EU-funded collaborative project between the EU and Canada called Smart Inclusive Living Environments (SMILE). The SMILE project is working to support aging in place using eHealth solutions with the aim of enabling older people to live an independent and active life, irrespective of frailty and physical or cognitive impairments, using new technologies developed with and for them. Throughout this review, references will be made to “older people” and “health technologies,” for this work, which refers to older people with frailty and impairment and new digital health technologies, respectively.

We conducted a scoping review following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) to synthesize knowledge; map existing evidence; and identify concepts, theories, sources, and knowledge gaps [24].

Eligibility Criteria

Inclusion Criteria

Studies were eligible for inclusion if they were published in English and in peer-reviewed journals, including conference papers, in any year as identified by our search strings. Eligible studies had to involve >65-year-old people with frailty and impairment. This includes individuals with cognitive decline or deterioration (eg, dementia), cognitive dysfunction (eg, aphasia), neurocognitive impairment, motor dysfunction (eg, stroke and ataxia), and physical and mental frailty or vulnerability. This information could be self-reported in the study. Furthermore, studies were eligible if perspectives, needs, and desires of older people with frailty and impairment were expressed and included in the development process of a digital health technology. Studies had to include a description of the development or design of digital health technologies. This aligned with the need to understand the methods of involvement in the design of digital health technology.

Exclusion Criteria

Table 1 provides an overview of the exclusion criteria. Review articles were excluded to avoid redundancy with respect to the original articles included in the review. Case reports, abstracts, and conference proceedings presenting preliminary data were excluded. Thus, full-text articles published with respect to conferences were not excluded.

The scope is limited to the use of digital health technologies meant to enable older people to age in place. Therefore, studies addressing the development of everyday technologies (eg, electrical appliances, technologies for indoor climate regulation, vacuum cleaners, jar openers, and electric curtains) in general products that are not used for specific health issues were excluded.

The following studies were also excluded: effect studies, such as those that only evaluated user experience or implementation (ie, acceptance, feasibility, effectiveness, and efficacy); and studies in which the methods of involvement were not reported or the involvement was not in the context of development of digital health technologies for aging in place.

Table 1. Exclusion criteria.
ExclusionLabelRemarks
Papers that have not been peer-reviewedNot peer-reviewed articleAlso include case reports, conference proceedings, and abstracts.
Full text not availableFull text not availableN/Aa
Exclude theoretical papers, that is, no involvement of population (eg, research protocols and theoretical papers)ProtocolsResearch (protocols) that is planned but not executed.
If data cannot be clearly identified for the age group or subgroup of >65 yearsPopulation not >65N/A
If data cannot be clearly identified for the group or subgroup of older people with frailty or impairmentPopulation not frail or impairedDefinition of frail and impaired; individuals with following disorders: cognitive decline (eg, dementia), cognitive dysfunction (eg, aphasia), neurocognitive impairment, motor dysfunction (eg, stroke, ataxia), frailty, vulnerability (not only social vulnerability).
Exclude studies that address the development of everyday technologies such as electrical appliances and technologies for indoor climate regulation and vacuum cleaners. In general, products not used for specific health issuesEveryday technologyThis group should be revisited after first round.
Studies in which methods cannot be clearly identifiedDoes not report methods of involvementDefinition of involvement: end user’s perspectives, needs, and desires are expressed and included in the development process to an extend beyond focus groups and classical participatory design.
Involvement not for the purposes of developmentInvolvement not for the purposes of developmentDefinition of development: development or design of new innovative technologies.
Exclude effect studiesEffect studiesArticles that only evaluate user experience or implementation, that is, acceptance, feasibility, effectiveness, efficacy, and so on.

aN/A: not applicable.

Information Sources and Search

Literature searches were performed in PubMed, Scopus, Embase, and IEEE. After the initial screening process, additional identification of relevant studies was performed using following two strategies: (1) identification of relevant studies in the reference lists of the screened studies and (2) input from experts in a workshop, thereby an additional 11 studies were identified; of these, 1 study met the inclusion criteria.

In PubMed, medical subject heading terms were included (search string for PubMed is listed in Multimedia Appendix 1), whereas in other databases (Scopus, Embase, and IEEE) keyword search was conducted (search strings for Scopus, Embase, and IEEE are listed in Multimedia Appendix 1). In total, 3 medical subject heading terms were included in the search string for PubMed: Telemedicine established in 1993, Cognitive Dysfunction established in 2012, and aged established in 1966.

Filters for the search strings included full-text availability and English language and excluded case reports, conference proceedings, abstracts, and nonpeer reviewed articles.

Screening

In total, 2675 studies were obtained using the search strings for PubMed, Scopus, Embase, and IEE; 922 (34.47%) studies were obtained using the search string for PubMed; 1753 (65.53%) was obtained from Scopus and 0 (0) from Embase and IEEE. A total of 29.35% (784/2675) duplicates were removed using Mendeley before importing to the Covidence database [25], a review software tool developed by the Cochrane Community. In Covidence, 1.12% (30/2675) more duplicates were removed based on the title, year, volume, and author. Duplicates were verified by the authors and removed, leaving 69.57% (1861/2675) studies for screening.

The studies were reviewed using Covidence [25]. After title and abstract screening, 63.1% (1688/2675) studies were excluded, with 10.25% (173/1688) studies then assessed for full-text eligibility. After full-text screening, 2.19% (37/1688) studies were included (Figure 1). During the data extraction phase of the review, 0.95% (16/1688) studies were further excluded because they did not meet the inclusion criteria (eg, only reporting on household appliances or design of future homes or lack of reporting on the involvement of older people in the design process). In addition, 1 study was identified by SMILE project partners in a workshop and was included in the review.

In total, 12.7% (22/173) articles were included. The screening process is illustrated in a PRISMA-ScR flow diagram (Figure 1). The first 4 authors conducted the initial screening by titles and abstracts, and the full-text screening was conducted on a first-to-come basis. All screenings (title, abstract, and full text) involved 4 authors, and each article screening included 2 reviewers. When in doubt about the eligibility of an article, all 5 authors discussed the evaluation. The full-text articles and those retrieved from the manual search and workshop were extracted by the authors EKW, LK, CW, and JMB. All authors participated in the synthesis and presentation of the findings.

Of the 22 studies, 9 (41%) studies had included some population aged <65 years. Because most of the population in these studies was >65 years of age, all the authors decided to include those studies and thus contributes important information about the design of new technologies for the age group of people >65 years.

Figure 1. PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) reporting flow diagram.

Overview

The following sections present the results of the data extraction. In the first section, a description of the general characteristics of the studies is provided. The second section presents the results related to the characterization of older people with frailty and impairment involved in the design processes and applied recruitment methods. The third section addresses the outcome of the study, the type of digital health technology developed, and contribution of older people with frailty and impairment to the design process. The fourth section addresses the methods and activities used to involve the older people, as well as time point of involvement during the development process.

General Characteristics of the Included Studies

The 22 studies included in this scoping review were published between 2009 and 2020, indicating that the practice of involving and focusing on how to involve older people with frailty and impairment in the design and development of digital health technologies is an increasing field. The included studies were not limited to a single geographic location. Geographic locations included the WHO Region of the Americas (5 studies from the United States, 1 from Chile, and 1 from Canada), the WHO European Region (2 studies from the United Kingdom; 3 from the Netherlands; and studies from Portugal, Germany, Italy, Finland, and Sweden), the WHO Western Pacific Region (1 study from Malaysia), and the WHO African Region (1 study from South Africa).

Table 2 provides an overview of the population groups included in these studies. Descriptions of the populations show a large representation of somatic conditions, cognitive conditions (eg, dementia and risk of cognitive decline), or a combination of both.

Table 2. Overview of conditions represented in studies (N=22).
Author and yearStudy population conditionValue, n (%)
Gövercin et al [26], 2010Risk of falling1 (5)
Hakobyan et al [27], 2015AMDa1 (5)
Willard et al [28], 2018Risk of cognitive decline1 (5)
Bogza et al [29], 2020Mild cognitive impairment1 (5)
Hassan et al [30], 2017
Kerkhof et al [31], 2019
Dementia2 (9)
de Barros et al [32], 2013
Wannheden and Revenäs [33], 2020
Parkinson disease2 (9)
Athilingam et al [34], 2017
Grossman et al [35], 2018
Wali et al [36], 2020
Patients with heart failure3 (14)
Greenhalgh et al [37], 2015
Jacelon et al [38], 2018
Macis et al [39], 2018
Albina and Hernandez [40], 2018
Multimorbidities, known health condition or long-term conditions4 (18)
Hoffman et al [41], 2019
Alvarez et al [42], 2020
Lehto et al [43], 2013
Oberschmidt et al [44], 2020
Pradhan et al [45], 2020
Vanoh et al [46], 2018
Du Preez and De La Harpe [47], 2019
Different known health conditions or not specified7 (32)

aAMD: age-related macular degeneration.

Recruitment Strategies

Table S3 (Multimedia Appendix 2) provides an overview of the recruitment strategies used and indicates studies in which no strategy was described. These include purposeful sampling, use of an outreaching approach (eg, attending local community support groups for the relevant study population), and invitation through partners. Moreover, the table provides an overview of the descriptions of the studied populations, including their characteristics and locations.

In 12 (55%) of the 22 studies, recruitment through partners was used to identify relevant and interested older people (for example, through patient associations) [26,28,31-33, 37,38,41,42,44,45,47]. In 4 studies, purposeful sampling was used to recruit participants [35,36,40,46], and in 2 studies, outreach approaches were applied. These included contact made through local support groups for people with age-related macular degeneration (AMD) [27] and through posters and written materials physically placed in local public areas and sent out electronically (ie, through social media) [30]. Finally, in 4 studies, the recruitment strategy was not described [29,34,39,43].

The population groups in the studies were recruited based on specific parameters, including age, health conditions, and living conditions. These parameters can affect the ways in which people can be accessed and recruited. For instance, older people with variations of cognitive decline, such as dementia, are often perceived as difficult to access and are included in the development of new technology. However, in the included studies, older people with dementia and AMD were recruited using different methods, including outreaching (2/22, 9%) [27,30] and through partners (1/22, 5%) [31]. This demonstrates that the otherwise difficult-to-access population groups could be approached and recruited using appropriate approaches, such as an outreaching approach, meaning that a combination of methods of approaching and recruiting participants to target different groups of older people can ensure a broader representation in the design process of digital health technology. However, the most used recruitment method is invitation through partners, in which there is a risk of bias. For example, Oberschmidt et al [44] problematize recruiting through partners and highlight participant bias as a study limitation. The study emphasizes that the older people who participated were very active and outgoing. Thus, it is not representative of all older people. This shows a gap in the knowledge on the use of outreaching approaches to recruit older people with frailty and impairment in the design of digital health technologies.

Outcome of Involvement

Table S4 (Multimedia Appendix 3) provides a short description of the aims of the studies, as well as an overview of how the included populations contributed to the study outcome. The final column describes the health technologies developed in each study.

In 21 (95%) of the 22 studies, older people lived at home; in 1 (5%) study, the population consisted of inpatients at a hospital [42]. Thus, most of the technologies developed are aimed for people living at home. The technologies developed in these studies include various digital and web-based solutions, such as applications and digital platforms (18/22, 82%) [27-36,38,39,41,42,44-47], assistive technologies (2/22, 9%) [37,40], wearables (1/22, 5%) [26], and interactive caring television (1/22, 5%) [43]. In 64% (14/22) of studies, different variations in outcomes were presented, including those involving participant contributions; for example, how involvement led to a list of themes to be considered when developing an app based on end user needs [36] and how inputs from patients were used to identify design requirements for the interface [35] and to develop a platform [43]. Older people also assessed accessibility, leading to 7 features being included in an apps to prevent delirium in hospitalized older people [42].

Of the 22 studies, 8 (36%) studies did not report or reflect on the participants’ contributions; that is 5 (23%) studies did not report participants’ contributions [28,39-41,47] and 3 (14%) studies described the participants’ contribution in evaluating a prototype or by how they are involved and not by their contribution to the development of the technology [27,31,46]. Thus, gaps in the consistency of description of the value of involvement and outcome of the specific contribution of the participants were identified.

Involvement Methods Used

Table S5 (Multimedia Appendix 4) provides an overview of the involvement methods used in the studies (eg, surveys and interviews). Moreover, it provides an overview of the time points when the methods were used to involve the participants in the design process, including needs identification, conceptualization, prototyping, or evaluation and further identifies whether the participants were included in one or several parts of the process.

The involvement of older people in this scoping review was assessed based on their involvement in 4 different parts of the development process. These four parts include the following: (1) needs identification, which is the first part of the development process in which end user needs are identified; (2) conceptualization, that is, the conceptualization of the final solution; (3) prototyping, that is, the development of a prototype; and finally, (4) evaluation of the prototype.

In 9% (2/22) of studies, participants were included in all 4 parts of the development process including, needs identification, conceptualization, prototyping, and final evaluation [32,34]. In 23% (5/22) of studies, participants were included in 3 parts of the development process [33,41-43,45], and in 23% (5/22) of studies, older people were included in 2 parts of the development process [27,28,31,35,37]. This overview shows that 12 (55%) of the 22 studies present a combined ecosystem of methods, with consecutive steps that aim to ensure the involvement of older people in different parts of the development process of a digital health technology, from the identification of needs to the generation of ideas, cocreation of a specific product, and final evaluation.

In 45% (10/22) of studies, participants were included in 1 part of the development process. In 60% (6/10) of these studies, the involvement was in the final part of the process, that is, the evaluation of the prototype, using a variety of different involvement methods including, focus groups, workshops, feedback sessions, questionnaire, assessment of acceptance, usability assessment, and rating scales [26,29,30,38,39,46]. Thus, there is a gap in knowledge of the means to involve older people with frailty and impairment in all parts of the design process, including the initial needs assessment phase.

The most frequently used method to involve older people in the included studies (10/22, 45%) was interviews. Interviews were used for needs identification, prototyping, and evaluation. In addition, workshops and focus groups were also commonly reported and applied in all 4 parts of the development process. The participants were mostly involved in the final part of the development process, the evaluation (15/22, 68% of the studies), and in the initial needs identification (13/22, 59% of the studies), whereas participants were least involved in the prototyping process (9/22, 41% of the studies) and conceptualization phase (6/22, 27% of the studies).

Finally, in 14% (3/22) of studies, specific theories were used to inform the analysis [36,44,46]. Du Preez and De La Harpe [47] applied a grounded theory methodology through an iterative and simultaneous process of data collection, coding, category development, and data comparisons to understand the perceptions of older people regarding technologies to support aging in place. Greenhalgh et al [37] position their study within “critical ethnography,” referring to phenomenological philosophy touching upon Maurice Jean Jacques Merleau-Ponty and Martin Heidegger’s work on perception. Finally, Pradhan et al [45] Used a constructivist grounded theory approach in their analysis. In total, 45% (10/22) of studies were conducted based on an existing framework or design concept (eg, feasibility study, scrum, PICTIVE [plastic interface for collaborative technology through video exploration] participatory design, and user-centered design framework) [26,27,31,32,35,36 ,38,39,42,46].


Overview

This scoping review sought to explore how older people with frailty and impairment are involved in various parts of the design processes of digital health technologies and to identify gaps or neglected steps in a user-involving design process. The focus has been on recruitment strategies, outcomes of involvement, and methods used to involve participants and address their perspectives, needs, and desires.

Principal Findings

In total, 3 gaps have been identified.

First, a gap in knowledge was identified regarding the use of different outreaching approaches to recruit older people with frailty and impairment in the design of digital health technologies. Involvement does not always begin during the recruitment process. Early involvement will enable an outreaching or alternative recruitment strategy to ensure a broad representation of participants and access hard-to-reach populations. An outreaching approach was effectively used in 2 studies that recruited older people with dementia [30] and AMD [27]. However, the most used recruitment strategy in the current literature is through partners or by purposeful sampling. Second, a gap was identified in relation to the description of the value of involvement and outcome of the specific contribution of the participants. Reflection on and description of the outcomes of participants’ contributions is important. Our findings show that some studies successfully reflected the outcome of participants’ contributions.

In one-third (7/22, 32%) of the studies, the specific outcome of the contribution is not reflected upon, leaving a gap in understanding the degree and value of the involvement process.

Third, a gap was identified in the knowledge regarding the means to involve participants in all parts of the design process, including the initial needs assessment phase. Using a variety of methods to involve older people with frailty and impairment in the design of new technologies is valuable, including focus groups, interviews, and workshops. An identified caution is the underrepresentation of involvement across the full design process as opposed to solely the final evaluation phase.

Involvement Starts With Recruitment

The findings indicate that choosing the right recruitment strategy is highly important to avoid recruitment bias and initiate a beneficial co-design process for older people with frailty and impairment. Therefore, reflecting on the use of different recruitment strategies is important to access a broader representation.

When recruiting participants, relevant factors should be considered, including how to reach the population of interest, as earlier studies have shown that older people and people with low digital skills are often left out or overlooked in the design process of new technologies. This lack of involvement can lead to increased inequity in health care services and a lack of access to new health technologies for those most in need [48].

The least commonly used recruitment strategy was the outreaching approach. The most used was purposeful sampling and invitation through partners. Oberschmidt et al [44] problematize recruiting through partners and highlight participant bias as a study limitation, emphasizing that the older people who participated were very active and outgoing. Future research need to focus on including older people who are less active and difficult to reach.

Hakobyan et al [27] benefited from using an outreaching approach to recruit people with AMD. The research group established contact with a support group for people with AMD. Over a period of 2 months, the research group attended 4 support group meetings to introduce themselves and learn more about their end users, including their capabilities and limitations. The research group found that the participants reluctance was sometimes related to their participation in research as an experimental subject, rather than an involved expert living with their specific condition. Together, the strategy to attend meetings for building relationships and obtaining a deeper insight into the reasons for the hesitation of potential participants ultimately enabled the research group to build a trusted relationship with the support group members, who eventually volunteered to participate in their study. Hassan et al [30] combined posters and written materials and distributed them physically and electronically (ie, via social media and email) to advertise the opportunity for involvement in the study. Using this method, approximately 25 people aged >65 years with dementia, memory problems, and mild cognitive impairments were recruited.

Finally, this scoping review found that some (4/22, 18%) studies that included older people with frailty and impairment had exclusion criteria that might have excluded relevant participants. These include cognitive, visual or hearing impairments, or severely limited dexterity in one or both hands [39], people with dementia [46], and those who required reading skills [38] or at least a secondary level of education (≥7 years) [46].

Description of Articulated Outcome of Involving Participants

The values of the involvement and contribution of the participants were explicitly addressed in 64% (14/22) studies. However, in 36% (8/22) studies, the contribution of involvement to the outcome was not described. This leaves a gap in the understanding of the degree and value of the involvement process.

The 14 studies addressed user involvement through a description of the involvement or reflection of the involvement. Athilingam et al [34] changed a prototype from being a chest-worn device used to monitor heart rate among patients with heart failure, to being a wrist-worn device, based on input from participants. In the study by Jacelon et al [38] the beta version of the user interface for “ASSISTwell,” a tablet app designed for older people to manage symptoms related to different chronic conditions, was developed using input from end users, retrieved through focus groups. De Barros et al [32] developed 4 apps for smartphone for the self-management of Parkinson disease, including (1) medication; (2) appointments; (3) my day, including disease status and symptoms; and (4) my data, including personal and health information, based on input retrieved through interviews, scoping sessions, focus groups, and usability testing with end users with Parkinson disease. Finally, Grossman et al [35] identified design requirements based on input from end users in the development of an interface to assist older people with heart failure.

In 36% (8/22) studies, the value of involvement and the contribution of the participants were not specified or reflected explicitly. Hakobyan et al [27] aimed through participatory and user-focused research to create a mobile assistive health care–related intervention for people with AMD to promote independent living. The methods used to involve the older people are described, including focus groups, observational studies, and design meetings, but the outcome of the involvement was not addressed. In other studies, participants were involved through surveys [40], interviews, and workshops [47] in the first part of the development process. When older people were involved in the initial and final aspects of design, this was often through interviews and observations [28], in 3 parts of the process including needs identification, prototyping, and evaluation of prototype [41], and in the final part of the design process [39], with no reflection on the outcome of involving the end users.

This could reflect the evolving nature of involving older people in the development of technologies or the lack of description of specific contributions. It could also reflect a lack of actual active involvement and use of specific input from older people; this is not clear in these cases. The number of studies that include older people in the design of technology increased between 2008 and 2020. However, the lack of inclusion of a broader older population, including those with disabilities, remains problematic [12]. Earlier studies found that involving older people in the design of technology has shown beneficial outcomes. These include learning about older people’s needs, adjusting technological designs according to older people’s needs, and a sense of participation among older people. This was emphasized by older participants, who appreciated being part of a generation that used technology [49]. A gap in knowledge about appropriate methods to involve people with disabilities and dementia in technology development remains [50], emphasizing the need for future work focusing on research that includes a broad variety of older people. Future research involving explicit reflections and descriptions could help the development of new ways to involve older people with frailty in the design of new technologies.

Involvement Methods Used Throughout the Design Process

The findings suggest that different methods, including focus groups, interviews, and workshops, to involve older people in the design of health technologies are valuable. An identified caution from these studies is the lack of involvement in the stages leading up to the final phase of the development process.

In most (15/22, 68%) studies, end users were involved in the final evaluation phase of the development process [26,28-35,38,39,41-43,46]. In 41% (9/22) studies, end users were involved in the third prototyping phase [27,31-34,37,41,42,45], and in 27% (6/24) studies, end users were involved in the second conceptualization phase [32-34,42,43,45]. Finally, 64% (14/22) studies included end users in the first phase “needs identification.” Thus, 36% (8/22) studies did not include end users in the initial “needs identification” of the development process [26,30,31,33,38,39,42,46]. This illustrates an overrepresentation of involvement in the final part of the design process conceptualization and evaluation, where a mix of focus groups, questionnaires, usability assessments, and observations are used. This is problematic considering the need for end user involvement to guide the initial development. Earlier studies suggest that involving end users, including people with dementia, can provide a better understanding of end users’ needs for a better design outcome and have a positive impact on future user experience [50]. Although there is no evidence in the studies stating that involvement in the beginning or the middle of a design process is especially rewarding, this review identifies a lack of involvement in the earlier and middle parts of design processes where needs and desires are normally identified before initiating the conceptualization and prototyping process.

In only 9% (2/22) of studies, participants were included in all 4 stages of the design process. Athilingam et al [34] involved participants with heart failure in initial needs identification through needs assessment interviews. Moreover, the participants answered a questionnaire and provided input to the conceptualization and feedback on the design, features, and ease of use during the development phase. Finally, a feasibility study was conducted leading to significant changes in the software and design, which changed from a chest-worn device to a wrist-worn device. These elements were all a part of the study that focused on “patient engagement” with the purpose of achieving both a well-targeted solution for this specific population group and achieving persistent self-care and self-management, including positive health behavior for this group. De Barros et al [32] included participants through interviews and a scoping session with focus on daily routines, motivation mechanisms, medication-related behaviors, and specific requests for the smartphone app. Furthermore, focus groups and usability testing were conducted throughout the development of a smartphone app for self-management in people with Parkinson disease.

Among the identified methods of involvement, there was no indication that some were more successful than the others in identifying older people’s needs. As the purpose of this review was not to judge how and which kind of involvement method have been beneficial for the outcome of the studies, the findings highlight the methods that may be used to involve older people in design processes, so that their needs and desires are heard. Newell et al [51] stated that classic standards and guidelines for user-centered design are not always appropriate for including older people and people with disabilities. This suggests that “user-sensitive inclusive design” is a new way of including older people. This includes forming a close bond with the participants and using experimental techniques to involve older adults; for example, through theatrical techniques using actors instead of personas to impersonate a diverse group of older disabled adults.

There was no general difference in how and when participants with different conditions are involved. People with cognitive impairment were involved through focus groups, interviews, and workshops and were involved in all parts of the process in various studies. This could indicate that specific diseases are not limited to one specific involvement method and that there are several possibilities in relation to the involvement of older people with frailty and impairment in development processes. However, as several studies failed to report the outcome of involvement, it is impossible to draw strong conclusions about the appropriateness of the specific methods used. Earlier studies address the need for improving traditional methods used to involve end users and to consider limitations related to an aging population, including activities such as interviews, questionnaires, and observations. Moreover, earlier findings suggest that involvement and engagement in the initial steps of a development process increase the potential to create a technology that considers relevant limitations and characteristics related to older people [52].

For future work, consideration of the outcome in relation to the degree of involvement is relevant for further assessment. This could include considering the level of involvement in relation to how many steps of the design process the end users are involved in and defining and assessing the degree of “active involvement” in using the different involvement methods. Further research on the effectiveness of these methods is required.

Recommendations from important stakeholders for engaging end users in the development of new medical and health technologies [20] have not been addressed in any of the included studies. There may be a need for specifications regarding how and in which steps user involvement and engagement should be performed. Our findings do not indicate a reason for excluding older people with frailty and impairment. It may be necessary for the regulatory bodies to clarify that these groups, if relevant to include, should not be excluded.

Limitations

Publications regarding digital health technology development are often conducted as part of the preparation for a certification process that is required by a funding body or in relation to a specific research goal. We may therefore have missed documentation in relation to commercial product development.

There may also be limitations in the representation of the specific disease addressed. The focus of this study was on older people with frailty and impairment, and thus, an overrepresentation of people with cognitive impairments is present, leaving out other major global disease burdens such as ischemic heart disease, diabetes, or multimorbidity, which may contribute to frailty.

Another limitation can be found in the definition of the applied technology used in this study. In this scoping review, studies involving welfare and mundane everyday technologies with no health-specific purpose were excluded, such as robot toilets, electrical curtains, robot vacuums, robot toilets, automated baths, and so on, as they are also part of a smart living environment and do not necessarily represent specific health technologies that were assessed in this study. Nevertheless, these studies may report the relevant methods on how to involve this population group for technology development, and the limitation to scope of the review is that we did not include those studies per the exclusion criteria.

Conclusions

This scoping review presents existing knowledge on how older people with frailty and impairment are involved in the design of digital health technologies that can contribute to their aging in place and also identifies gaps or neglected steps in a user-involving design process.

A gap in knowledge was identified regarding the use of outreaching approaches to recruit older people with frailty and impairment in the design of digital health technologies. The most commonly used recruitment strategy in the current literature is recruitment through partners or by purposeful sampling. The risk of bias in selecting participants is higher when using these forms of recruitment than when using an outreaching approach. However, it is important to emphasize that the literature does not suggest how the outcome of studies is affected by the different strategies.

Another gap was identified in the description of the value of involvement and the outcome of the specific contribution of the participants. Thus, reflection on the use of different involvement methods in future work could help evolve the existing practices and enable more older people, who are not commonly included in development processes, to take part in future projects.

Finally, a series of different methods used to involve older people in the development of digital health technologies was identified. However, a gap was identified in the knowledge regarding the means to involve the older people in all parts of the design process, including the initial needs assessment phase. The literature does not imply which part of the development process involvement is most beneficial. However, only few studies included participants throughout the development process, and an overrepresentation of participants involved at the end of the design process and underrepresentation of participants involved in the first steps of the design process were identified.

Acknowledgments

This research was funded by the European Union’s Horizon 2020 Research and Innovation Program under grant 101016848 and the Canadian Institutes of Health Research.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Search strings.

PDF File (Adobe PDF File), 49 KB

Multimedia Appendix 2

Table S3. Recruitment strategies.

PDF File (Adobe PDF File), 113 KB

Multimedia Appendix 3

Table S4. Outcome of the study.

PDF File (Adobe PDF File), 100 KB

Multimedia Appendix 4

Table S5. Involvement methods.

PDF File (Adobe PDF File), 107 KB

  1. Ageing and health. World Health Organization.   URL: https://www.who.int/news-room/fact-sheets/detail/ageing-and-health [accessed 2021-11-01]
  2. Healthy places terminology. Centers for Disease Control and Prevention.   URL: https://www.cdc.gov/healthyplaces/termino logy.htm [accessed 2021-11-01]
  3. Cagnin C, Amanatidou E, Keenan M. Orienting European innovation systems towards grand challenges and the roles that FTA can play. Sci Public Policy 2012 Apr 02;39(2):140-152. [CrossRef]
  4. De Smedt P, Borch K, Fuller T. Future scenarios to inspire innovation. Technol Forecast Social Change 2013 Mar;80(3):432-443. [CrossRef]
  5. Sixsmith A. Technologies for Active Aging. New York: Springer; 2013.
  6. Östlund B, Olander E, Jonsson O, Frennert S. STS-inspired design to meet the challenges of modern aging. Welfare technology as a tool to promote user driven innovations or another way to keep older users hostage? Technol Forecast Social Change 2015 Apr;93:82-90. [CrossRef]
  7. What is digital health? U.S. Food & Drug Administration.   URL: https://www.fda.gov/medical-devices/digital-health-center -excellence/what-digital-health [accessed 2021-10-21]
  8. Lindsay S, Brittain K, Jackson D, Ladha C, Ladha K, Olivier P. Empathy, participatory design and people with dementia. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2012 Presented at: CHI '12: CHI Conference on Human Factors in Computing Systems; May 5 - 10, 2012; Austin Texas USA. [CrossRef]
  9. Harrington C, Martin-Hammond A, Bray K. Examining identity as a variable of health technology research for older adults: a systematic review. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022 Presented at: CHI '22: CHI Conference on Human Factors in Computing Systems; Apr 29 - May 5, 2022; New Orleans LA USA. [CrossRef]
  10. Peek ST, Wouters EJ, van Hoof J, Luijkx KG, Boeije HR, Vrijhoef HJ. Factors influencing acceptance of technology for aging in place: a systematic review. Int J Med Inform 2014 Apr;83(4):235-248 [FREE Full text] [CrossRef] [Medline]
  11. Travers J, Romero-Ortuno R, Bailey J, Cooney M. Delaying and reversing frailty: a systematic review of primary care interventions. Br J Gen Pract 2018 Dec 03;69(678):e61-e69. [CrossRef]
  12. Harrington MM, Daniele E. Gerontology: Changes, Challenges, and Solutions, Volume 2. Santa Barbara, California, United States: ABC-CLIO, LLC; 2016.
  13. Neven L. 'But obviously not for me': robots, laboratories and the defiant identity of elder test users. Sociol Health Illn 2010 Feb 01;32(2):335-347 [FREE Full text] [CrossRef] [Medline]
  14. Bergschöld JM, Neven L, Peine A. DIY gerontechnology: circumventing mismatched technologies and bureaucratic procedure by creating care technologies of one's own. Sociol Health Illn 2020 Feb;42(2):232-246 [FREE Full text] [CrossRef] [Medline]
  15. Giaccardi E, Kuijer L, Neven L. Design for resourceful ageing: intervening in the ethics of gerontechnology. In: Proceedings of the Future Focused Thinking - DRS International Conference 2016. 2016 Presented at: Future Focused Thinking - DRS International Conference 2016; Jun 27 - 30, 2016; Brighton, United Kingdom. [CrossRef]
  16. Botero A, Hyysalo S. Ageing together: steps towards evolutionary co-design in everyday practices. CoDesign 2013 Mar;9(1):37-54. [CrossRef]
  17. Bergschöld J. Configuring dementia; how nursing students are taught to shape the sociopolitical role of gerontechnologies. Front Sociol 2018 Apr 9;3 [FREE Full text] [CrossRef]
  18. Guidance document patient engagement in the design and conduct of medical device clinical studies. U.S. Food & Drug Administration.   URL: https:/​/www.​fda.gov/​regulatory-information/​search-fda-guidance-documents/​patient-engagement -design-and-conduct-medical-device-clinical-studies [accessed 2022-01-26]
  19. ISO 9241-210:2010(en) Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems. ISO. 2010.   URL: https://www.iso.org/obp/ui/#iso:std:iso:9241:-210:ed-1:v1:en [accessed 2021-10-15]
  20. Campbell B, Sedrakyan A. Patient involvement in regulation: an unvalued imperative. Lancet 2021 Jun 05;397(10290):P2147-P2148.
  21. mHealth: use of appropriate digital technologies for public health: report by the Director-General. World Health Organization. 2017.   URL: https://apps.who.int/iris/handle/10665/274134 [accessed 2021-09-20]
  22. Gray L, Bernabei R, Berg K, Finne-Soveri H, Fries B, Hirdes J, et al. Standardizing assessment of elderly people in acute care: the interRAI Acute Care instrument. J Am Geriatr Soc 2008 Mar;56(3):536-541 [FREE Full text] [CrossRef] [Medline]
  23. Bitar H, Alismail S. The role of eHealth, telehealth, and telemedicine for chronic disease patients during COVID-19 pandemic: a rapid systematic review. Digit Health 2021 Apr 19;7:20552076211009396 [FREE Full text] [CrossRef] [Medline]
  24. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018 Oct 02;169(7):467-473 [FREE Full text] [CrossRef] [Medline]
  25. Better systematic review management. Covidence.   URL: https://www.covidence.org/ [accessed 2021-02-22]
  26. Gövercin M, Költzsch Y, Meis M, Wegel S, Gietzelt M, Spehr J, et al. Defining the user requirements for wearable and optical fall prediction and fall detection devices for home use. Inform Health Soc Care 2010 Dec 06;35(3-4):177-187. [CrossRef] [Medline]
  27. Hakobyan L, Lumsden J, O'Sullivan D. Participatory design: how to engage older adults in participatory design activities. Int J Mob Hum Comput Interact 2015;7(3).
  28. Willard S, Cremers G, Man YP, van Rossum E, Spreeuwenberg M, de Witte L. Development and testing of an online community care platform for frail older adults in the Netherlands: a user-centred design. BMC Geriatr 2018 Apr 07;18(1):87 [FREE Full text] [CrossRef] [Medline]
  29. Bogza L, Patry-Lebeau C, Farmanova E, Witteman HO, Elliott J, Stolee P, et al. User-centered design and evaluation of a web-based decision aid for older adults living with mild cognitive impairment and their health care providers: mixed methods study. J Med Internet Res 2020 Aug 19;22(8):e17406 [FREE Full text] [CrossRef] [Medline]
  30. Hassan L, Swarbrick C, Sanders C, Parker A, Machin M, Tully MP, et al. Tea, talk and technology: patient and public involvement to improve connected health 'wearables' research in dementia. Res Involv Engagem 2017;3:12 [FREE Full text] [CrossRef] [Medline]
  31. Kerkhof Y, Pelgrum-Keurhorst M, Mangiaracina F, Bergsma A, Vrauwdeunt G, Graff M, et al. User-participatory development of FindMyApps; a tool to help people with mild dementia find supportive apps for self-management and meaningful activities. Digit Health 2019 Mar 26;5:2055207618822942 [FREE Full text] [CrossRef] [Medline]
  32. de Barros A, Cevada J, Bayés À, Alcaine S, Mestre B. User-centred design of a mobile self-management solution for Parkinson's disease. In: Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia. 2013 Presented at: MUM '13: 12th International Conference on Mobile and Ubiquitous Multimedia; Dec 2 - 5, 2013; Luleå Sweden. [CrossRef]
  33. Wannheden C, Revenäs Å. How people with Parkinson's disease and health care professionals wish to partner in care using eHealth: co-design study. J Med Internet Res 2020 Sep 21;22(9):e19195 [FREE Full text] [CrossRef] [Medline]
  34. Athilingam P, Clochesy JM, Labrador MA. Intervention mapping approach in the design of an interactive mobile health application to improve self-care in heart failure. Comput Inform Nurs 2018 Feb;36(2):90-97. [CrossRef] [Medline]
  35. Grossman L, Feiner S, Mitchell E, Masterson Creber R. Leveraging patient-reported outcomes using data visualization. Appl Clin Inform 2018 Jul 01;9(3):565-575 [FREE Full text] [CrossRef] [Medline]
  36. Wali S, Keshavjee K, Nguyen L, Mbuagbaw L, Demers C. Using an electronic app to promote home-based self-care in older patients with heart failure: qualitative study on patient and informal caregiver challenges. JMIR Cardio 2020 Nov 09;4(1):e15885 [FREE Full text] [CrossRef] [Medline]
  37. Greenhalgh T, Procter R, Wherton J, Sugarhood P, Hinder S, Rouncefield M. What is quality in assisted living technology? The ARCHIE framework for effective telehealth and telecare services. BMC Med 2015 Apr 23;13:91 [FREE Full text] [CrossRef] [Medline]
  38. Jacelon C, LeBlanc R, Alkhawaldeh M, Ridgway J, Marquard J, Choi J. Developing ASSISTwell, a tablet application to support older adult’s self-management of symptoms of chronic conditions. Gerontechnology 2018 Mar 26;17(1):18-28.
  39. Macis S, Loi D, Ulgheri A, Pani D, Solinas G, Manna SL, et al. Design and usability assessment of a multi-device SOA-based telecare framework for the elderly. IEEE J Biomed Health Inform 2020 Jan;24(1):268-279. [CrossRef]
  40. Albina E, Hernandez A. Assessment of the elderly on perceived needs, benefits and barriers: inputs for the design of intelligent assistive technology. In: Proceedings of the 2018 16th International Conference on ICT and Knowledge Engineering (ICT&KE). 2018 Presented at: 2018 16th International Conference on ICT and Knowledge Engineering (ICT&KE); Nov 21-23, 2018; Bangkok, Thailand. [CrossRef]
  41. Hoffman AS, Bateman DR, Ganoe C, Punjasthitkul S, Das AK, Hoffman DB, et al. Development and field testing of a long-term care decision aid website for older adults: engaging patients and caregivers in user-centered design. Gerontologist 2020 Jul 15;60(5):935-946 [FREE Full text] [CrossRef] [Medline]
  42. Alvarez E, Garrido M, Ponce D, Pizarro G, Córdova A, Vera F, et al. A software to prevent delirium in hospitalised older adults: development and feasibility assessment. Age Ageing 2020 Feb 27;49(2):239-245. [CrossRef] [Medline]
  43. Lehto P. Interactive CaringTV® supporting elderly living at home. Australas Med J 2013 Sep 02;6(8):425-429. [CrossRef]
  44. Oberschmidt K, Grünloh C, Tunç S, van Velsen L, Nijboer F. You can’t always get what you want: streamlining stakeholder interests when designing technology-supported services for Active and Assisted Living. In: Proceedings of the 32nd Australian Conference on Human-Computer Interaction. 2020 Presented at: OzCHI '20: 32nd Australian Conference on Human-Computer-Interaction; Dec 2 - 4, 2020; Sydney NSW Australia. [CrossRef]
  45. Pradhan A, Jelen B, Siek K, Chan J, Lazar A. Understanding older adults' participation in design workshops. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020 Presented at: CHI '20: CHI Conference on Human Factors in Computing Systems; Apr 25 - 30, 2020; Honolulu HI USA. [CrossRef]
  46. Vanoh D, Ishak IH, Shahar S, Manaf ZA, Ali NM, Noah SA. Development and assessment of a web-based intervention for educating older people on strategies promoting healthy cognition. Clin Interv Aging 2018;13:1787-1798 [FREE Full text] [CrossRef] [Medline]
  47. Du Preez V, De La Harpe R. Engaging aging individuals in the design of technologies and services to support health and well-being: constructivist grounded theory study. JMIR Aging 2019 Mar 20;2(1):e12393 [FREE Full text] [CrossRef] [Medline]
  48. Winkle B, Carpenter N, Moscucci M. Why aren't our digital solutions working for everyone? AMA J Ethics 2017 Nov 01;19(11):1116-1124 [FREE Full text] [CrossRef] [Medline]
  49. Fischer B, Peine A, Östlund B. The importance of user involvement: a systematic review of involving older users in technology design. Gerontologist 2020 Sep 15;60(7):e513-e523 [FREE Full text] [CrossRef] [Medline]
  50. Suijkerbuijk S, Nap HH, Cornelisse L, IJsselsteijn WA, de Kort YA, Minkman MM. Active involvement of people with dementia: a systematic review of studies developing supportive technologies. J Alzheimers Dis 2019 Jun 18;69(4):1041-1065. [CrossRef]
  51. Newell AF, Gregor P, Morgan M, Pullin G, Macaulay C. User-sensitive inclusive design. Univ Access Inf Soc 2010 Jul 11;10(3):235-243. [CrossRef]
  52. Duque E, Fonseca G, Vieira H, Gontijo G, Ishitani L. A systematic literature review on user centered design and participatory design with older people. In: Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems. 2019 Presented at: IHC '19: XVIII Brazilian Symposium on Human Factors in Computing Systems; Oct 22 - 25, 2019; Vitória Espírito Santo Brazil. [CrossRef]


AMD: age-related macular degeneration
EU: European Union
PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews
SMILE: Smart Inclusive Living Environments
WHO: World Health Organization


Edited by A Kushniruk; submitted 07.03.22; peer-reviewed by Y Hershcovitz, C Ganoe, M Ehn; comments to author 14.05.22; revised version received 08.07.22; accepted 20.11.22; published 27.01.23

Copyright

©Emilie Kauffeldt Wegener, Jenny M Bergschöld, Carly Whitmore, Marjolein Winters, Lars Kayser. Originally published in JMIR Human Factors (https://humanfactors.jmir.org), 27.01.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on https://humanfactors.jmir.org, as well as this copyright and license information must be included.