Designing eHealth Applications to Reduce Cognitive Effort for Persons With Severe Mental Illness: Page Complexity, Navigation Simplicity, and Comprehensibility

Introduction

The closing of long-term hospitals and the increasingly abbreviated stays in acute care facilities leave persons with severe mental illness (SMI) more dependent on outpatient services and self-care to support their community well-being. Prevailing in-person mental health service delivery models have resulted in a situation where many treatments, proven efficacious over the past 30 years, are not readily available [1-4]; available services rarely meet the established standards for care [5-8]; only 40%-50% receive specialty mental health care in a given year [6-8]; and only 4%-15% receive even minimally adequate mental health treatment—far short of evidence-based standards [7,9,10]. Consequently, the majority of persons with SMI live in communities where they and their families receive few services, and those services fall substantially short of the established standards for care [11,12]. Clearly there is a need for cost-effective ways to increase receipt of care and successfully disseminate evidence-based interventions to communities [11].

eHealth technologies are being used increasingly in general medical care [13-16] and to a lesser, though growing, extent in mental health treatment [17-20] as a way to improve service receipt and reduce illness burden. In this study, eHealth refers to the use of consumer-facing information and communication technologies to support health [21,22], with a focus on Web-based and mobile phone apps. As eHealth technologies become more prominent, the dearth of models to effectively design them for persons with SMI [21,22], and others with cognitive impairments, will result in significant obstacles to obtaining services for these individuals and exacerbate already inadequate receipt of services [6]. Several investigators have identified the difficulty that those with SMI have in using websites designed for the general public [22-24]. In response, research has been conducted to develop designs appropriate for those with SMI [25-28].

As part of our prior work, we developed an empirically supported design model to create eHealth technologies that persons with SMI, with little or no prior technology experience, could use effectively [23]. The resulting nascent model, termed the Flat Explicit Design Model (FEDM), proved to be quite effective. Using this model, we created a Web-based intervention, termed Schizophrenia Online Access to Resources (SOAR), to provide in-home multifamily psychoeducational treatment to persons with schizophrenia and their family members [23]. The intervention website was highly valued, frequently used, and had significant effects on important outcomes (eg, reducing positive symptoms) [29,30]. Its design proved to be quite effective. In tests comparing this website to public websites for persons with SMI, those with SMI took less time to find contents, had greater success finding contents, and rated it easier and less frustrating to use [24].

This study evaluated several additions to the design model (FEDM, Textbox 1) used to create SOAR. Observations during previous usability studies suggested that some individuals with SMI had difficulty scanning a page effectively for content and creating a mental model of the layout and organization of a screen’s contents [23,28]. Generally, these are important requirements to effectively navigate standard eHealth apps. Others have found that individuals with schizophrenia are not able to use websites that are well designed, but intended for the general public [22]. These observations, coupled with previous findings, and knowledge of the cognitive deficits associated with SMI led us to develop the novel concept of page complexity as potentially important to the cognitive effort required to comprehend and effectively use an eHealth app. In elaborating a model of page complexity, we developed the new design constructs of distinct topic areas and navigation areas. Additionally, our experience led us to speculate that the number of columns used to organize contents, number of words and hyperlinks on a page, and the text’s reading level and simplicity (ie, reading ease) would also influence the cognitive effort required for comprehension and navigation and thus, an app’s usability. We hypothesized that these constructs and variables would have an effect on the ability of persons with SMI to navigate eHealth technologies. This study was designed to test: these new constructs; the initial validity of our concept of page complexity; a set of variables to define page complexity; the addition of a new variable, reading ease, to the FEDM; and three functional dimensions we developed to organize the 19 variables of the FEDM. These dimensions were designed to have practical implications for creating effective designs. The study had persons with schizophrenia perform tasks on 5 public mental health websites during which data were collected on the ability to find contents, time to find contents, subjective evaluation of the websites, and influence of the homepage designs on a user’s understanding of a website.

Methods

Participant Recruitment and Selection Criteria

Participants were recruited from 6 community psychiatric rehabilitation outpatient centers, which provided day treatment programs. To receive services, one had to have an SMI diagnosis and appropriate insurance (eg, Medicare). Staff identified individuals with proper diagnoses and discussed the study with them. Those who were interested were told when study staff would be there (ie, to conduct usability testing). Enrollment criteria were as follows: age ≥18 years; a chart diagnosis of schizophrenia or schizoaffective disorder; physical ability to read the screen of a computer and use a mouse; and ability to read at a 5th grade level. There were no requirements for prior computer, Internet, or website use. The research protocol was approved by the University of Pittsburgh’s institutional review board.

Participant Background Information

The Global Assessment Scale (GAS) [31] was used by study staff who conducted the usability testing to rate each participant’s level of functioning. All staff were formally trained and experienced at rating persons with schizophrenia using the GAS. Sociodemographic and computer experience information were also collected (Table 1) —one participant did not answer the computer access question.

Choice of Websites for Evaluation

Five websites that included information on schizophrenia were chosen for the study. The website SOAR [30] was developed to deliver family psychoeducational treatment using the FEDM. Its address was not provided because it is currently under study. The other 4 were identified using Google to search for the word schizophrenia and by searching the Google directory. The websites were chosen to represent variation on important design variables:Schizophrenia.com (started in 1995), one of the first websites listed in the search results, provides support groups, science-based information, and discussion forums for individuals and families; the website of the National Alliance on Mental Illness (started in 1979), designed to provide education, advocacy, help, and public awareness to individuals and family members; Chovil.com (started in 1997), was created by an individual with schizophrenia and, thus, may contain design insights not commonly found in other sites; and the National Mental Health Association-Mental Health America (founded in 1909) has a website that focuses on individuals with mental illness and provides resources to promote mental health wellness. All websites were downloaded and cached locally to ensure that their contents remained static throughout the study.

Training of Testers

All testers were mental health professionals and had experience working with individuals with SMI. They also all had experience in providing training to individuals with schizophrenia on how to use computers and navigate websites from previous research projects. Two of the authors (AR and SE) provided training to testers and supervised them as they conducted the testing procedures on each other and other staff. Additional supervision was provided during the data collection for this study.

Participant Training and Testing

To ensure the basic knowledge needed to navigate the websites, each participant was taken through a brief tutorial using 3 preselected public websites (different from the 5 tested) to teach the basic elements that would be needed to navigate the 5 websites being tested. The topics covered included using a mouse, hyperlink text, navigation buttons, pop-up menus, and scroll bar to scroll up and down a page. The longest training took 22 minutes. This was based on tutorials we had used previously [29,32]. All who met eligibility criteria were able to master the use of these basic elements. Training was provided before the testing occurred by the research staff conducting the usability testing.

Characterizing Each Website’s Homepage Complexity

Each website’s homepage was characterized in terms of 19 variables (Table 2). Reading level (in terms of school year), reading ease (for which higher values represent easier reading ease), and total number of words were determined using the grammar (which utilized the Flesch and Kincaid formulas) [33] and word count functions in Microsoft Word. The number of hyperlinks and columns of contents on the homepage were counted manually. If a homepage had sections with different numbers of columns, a weighted average was calculated by summing the number of columns in each section multiplied by a section’s proportion to the total length of the page. A topic area was defined as being a distinct area of a screen or page that was devoted to a given topic or purpose, and it could contain both nonhyperlink and hyperlink text. A navigation area was a distinct area of a screen or page that contained 1 or more hyperlinks grouped together by proximity and typically related to one another via a common topic. Many topic areas were composed of text (ie, nonhyperlink text), which introduced the topic of the area, and included 1 or more hyperlinks associated with the topic. If a set of hyperlinks occupied a separate area of the page, was not associated with any text, and represented a distinct topic from other areas in proximity, it was classified as a topic area, and it was also classified as a navigation area.

Website Evaluation Procedures

The usability testing occurred in private rooms at the psychiatric rehabilitation outpatient centers. Testing was done on our computers. During the testing, to control for learning effects, the order of testing of the 5 websites was varied across participants using a 5×5 Latin square design. Likewise, the order of presentation of tasks within each of the 3 different types of tasks participants performed were varied across individual websites.

Website usability testing was conducted using software, Ergobrowser, (Ergosoft) for recording and timing website navigation. Participants were timed by pressing a special keyboard button, which started and stopped the timing for each subtask. It also recorded which subtask participants were performing.

Tasks to Evaluate Website Usability Performance

Task 1: Homepage Content Disclosure Performance

To assess the effectiveness with which a homepage’s design conveys information about a website’s contents (what we term homepage content disclosure), participants were asked to study a homepage for 60 seconds. They could scroll down the page but not use the mouse to open pull-down or pop-up menus. Then, while still being able to look at the page, they decided (using the options of yes or no) whether they thought a website contained information on 7 topics: how schizophrenia is treated, how to find a good psychiatrist, the side effects of medications, the causes of schizophrenia, how to find volunteer work, how to know whether a person has schizophrenia, and how many people have schizophrenia. The 7 topics were chosen so that information was present on 5 of the 7 topics on each website. For analyses the participants were put into one of 2 groups based on the number they got correct: one group was those who got ≤5 correct, the other group was those who got >5 correct. One subject did not complete this task.

Task 2: Find Specific Information—Ability and Time

To assess the ability to accurately navigate each website, participants were asked to find content on 3 topics: (1) treatments for schizophrenia, (2) side effects of medications used to treat schizophrenia, and (3) the causes of schizophrenia. Participants were given a maximum of 5 minutes to find the information on each topic. Once a search was over, participants were returned to the homepage to begin the next search. For analysis, we used 2 measures: whether for all 5 tasks the subjects found the correct information and, for the tasks solved correctly, the mean number of seconds needed to correctly find the information. This time was missing for 5 participants.

Task 3: Reactions to a Website

Following testing on a website, participants were asked to rate their impressions of the ease of use of a website using a 5-point scale ranging from 1 (“Not at all”) to 5 (“Extremely”).

Data Analyses

The values obtained for the number of a website’s navigation and topic areas, links, columns and words on the home page, and home page length are included in Table 2. Each variable was dichotomized; that is, the values of each variable were coded into 2 levels: “less complex” versus “more complex” (Table 3). For the 5 websites used in this study, several of the 19 variables overlapped in their variation with other variables, yielding 12 distinct variable groups (Table 3).

We organized the 19 variables of the FEDM into 3 dimensions or factors. To make an initial assessment of the validity of these 3 dimensions, coefficient alphas were computed to assess the internal consistency of the variables within the 3 dimensions. A summary score was created for each of the 3 dimensions by adding the dichotomized values of the variables within each dimension. Analyses of the relationships of the usability outcomes to the 19 variables, as well as to the 3 dimensions’ summary scores, were performed using mixed model regressions that accounted for the repeated measures within subjects and across websites. Logistic regressions were used for ability to find all 3 pieces of information versus not finding all 3, and the content disclosure task, which consisted of the number of times each participant correctly identified whether 7 topics were addressed by a website, after only examining the homepage. Performance for the content disclosure task was grouped into ≤5 or >5. Linear regressions were used for mean time to find information, and ease of use. Initially univariate regressions were done followed by multivariable models that included all 12 nonoverlapping FEDM variable groups. Analyses of the summary dimension scores were also completed with mixed effect regressions.

Demographic information, reports of computer experience, and overall function measured by the GAS were summarized with means for continuous variables and frequencies for categorical variables.

Results

Principal Findings

The mean age of participants was 47.2 (SD=6.62); 50% (19/38) were females, and 40% (15/38) were African American/Black (Table 1). Twenty participants (53%) reported prior computer use, of which 27% (10/38) had home access, 27% had only nonhome access, and 50% (19/38) had used websites previously.

Characteristics of the Websites’ Design Complexity

Table 2 presents each of the 5 websites characterized according to the 19 variables of the FEDM. In terms of dimension 1, page complexity, Chovil and SOAR have the lowest design complexity, followed by Schizophrenia.com, with the National Alliance on Mental Illness (NAMI) and the National Mental Health Association (NMHA) having the most complex designs. For example, Chovil has the fewest columns, highest reading ease, and fewest hyperlinks. SOAR is designed using the fewest navigation areas, and both have the fewest topic areas for presenting contents. For dimension 2, navigation simplicity, SOAR and SZ.com have the simplest designs. The other 3 websites are relatively complex on this dimension. With dimension 3, comprehensibility, all websites have some good design characteristics. SOAR has the most, and is the only one to employ the target audience’s dialect and require relatively less inference to understand contents. Chovil has the lowest reading level, followed by Schizophrenia.com, NMHA, and then SOAR, with NAMI having the highest.

Ability and Time to Find Information, Subjective Ease of Use Ratings, and Content Disclosure Task

Univariate analyses showed that the following 11 variables were predictors of the 4 usability outcomes (content disclosure, ability and time to find contents, ease of website use): few navigation areas,topic areas, columns, shallow depth of hierarchy, few themes, few display distractions, easy to moderate reading ease, constant navigational toolbar, moderate reading level, and explicit hyperlink labels (Table 3). In addition to the aforementioned 11 variables, the ability of a homepage to convey information about a website’s contents to users (ie, content disclosure) was also associated with fewer hyperlinks. The ability to find the 3 pieces of information and the time to find the information were also significantly associated with the following variables: the use of low-level hyperlink categories, having hyperlinks in the upper left of the page, having fewer words on a homepage, shorter homepage length, use of dialect, and minimal inference required. In addition to the aforementioned 11 variables, ease of use ratings were also associated with fewer hyperlinks, use of low-level hyperlink categories, having hyperlinks in the upper left of the page, having the introductory text located after the hyperlinks to contents, having fewer words on a homepage, shorter homepage length, use of dialect,and minimal inference required.

The multivariable analyses indicated that the ability to navigate a website and find the information were significantly related to shallow depth, few themes presented, few display distractions, easy to moderate reading ease (odds ratio, OR=24.5, P<.001) and fewer hyperlinks (OR=–0.08, P<.001). For the mean time to find the information, shallow depth, moderate reading level (68 vs 47 seconds, P<.001) and fewer hyperlinks (68 vs 81 seconds, P<.01) remained significant. Users’ ease of website use ratings were significantly related to having fewer navigational areas, topic areas, and columns (3.3 vs 2.9, P=.06), shallow depth, few themes presented, few display distractions, easy to moderate reading ease (3.6 vs 2.9, P<.001) and fewer hyperlinks (2.8 vs 1.6, P<.001). For the content disclosure task, the ability of users to correctly identify the contents of a website after only viewing the homepage was significantly and positively influenced by shallow depth, moderate reading level (OR=3.9, P=.001) and having fewer hyperlinks (OR=–3.4, P<.001).

Assessment of the Theoretical Dimensions of the FEDM: Page Complexity, Comprehensibility, and Navigation Simplicity

The summed scores for the 3 proposed dimensions all had coefficient alphas=0.8 (page complexity alpha=0.95, Navigational Simplicity alpha=0.80, and comprehensibility alpha=0.84), indicating that the variables within each of these dimensions are internally consistent. For the usability outcomes of ability to find information, mean time to find information, and ease of website use scores, the dimensions page complexity and navigational simplicity were significantly related to more positive outcomes (P<.01 for all). For the fourth usability outcome of correctly identifying which information is present in a website after studying the homepage, the summary scores of Navigational Simplicity (P<.001) and comprehensibility (P=.02) were significant.

Discussion

Summary of Evidence

This study extends prior work with the FEDM for creating eHealth apps for individuals with SMI [23,24]. These findings indicate that the novel design variables of topic areas, navigation areas, and columns influenced the usability of websites for people with schizophrenia. The study provides support that the 19 variables in the FEDM capture important design constructs that influence usability. This evaluation provides initial support for the validity of the 3 new dimensions we created for organizing these 19 variables, that of page complexity, navigation simplicity, and comprehensibility. These dimensions focus on 3 characteristics of a design that are critical to the usability of technology, and likely the associated cognitive effort required by individuals for its use.

The written text of the tested websites was at a relatively high reading level. The calculated reading levels were potentially higher than their effective levels due to a number of words that increased the reading levels but that participants likely understood, such as “schizophrenia.” In addition, the introductory text on the homepage of SOAR was at the bottom of the page, after the hyperlinks to contents, and thus its reading level likely had less of an effect on the complexity of the page for navigation, compared with websites where the text came prior to the hyperlinks or was interspersed among the hyperlinks.

For the variable “reading ease,” low to intermediate levels were the best. Our common finding has been that, for most variables, less complexity is the best, but only within the context of certain trade-offs, as discussed later. This finding may indicate that text that is too simple may be written using more words, which makes it less efficient and more difficult to understand. Text at a slightly more advanced level may be more efficient and consequently easier for many to understand, including people with schizophrenia. Additionally, the amount of text was associated with the ability to find information, time to find information, and ease of use ratings. The lowest quantity of text was the best. Our prior research indicates that more words should be used when necessary to ensure comprehension and make meanings explicit (eg, for hyperlinks) [24]. Taken together, this group of findings is consistent with the conclusion that text efficiency is the critical variable, as long as understanding is not compromised by using wording that is too brief, at too high of a reading level, or too abstract and, consequently, not explicit.

The number of columns used to organize the information on a screen may not capture the full impact of this variable on usability. Some websites varied the number of columns used as one proceeded down a screen, whereas others were consistent throughout the screen. It is likely that information displays that change over a single screen or page require more cognitive effort to understand and navigate than those that are consistent. In prior work we found that some persons with SMI did not understand the convention that columns separate contents across rows in a table[23]. Thus, the use of columns, and other geometric conventions, can pose usability obstacles.

When designing an app, several trade-offs must be made between the dimensions of the FEDM. For example, there is a trade-off between information density and a page’s complexity. A standard approach to design is to minimize the content on a given screen, by having short, one- or two-word-long hyperlinks, relatively few hyperlinks on a screen, minimal text on navigation pages (vs final destination pages), and no more information than can fit on a single screen (ie, scrolling is not used). This approach emphasizes “paging,” going to a new page to find additional information or hyperlinks, where again the contents are minimal. This minimizes information density and a page’s layout complexity. These are 2 strengths of this design, but there are also potential shortcomings with this approach, particularly for those with SMI: (1) it requires relatively more navigation; thus, the difficulty of searching through an individual page is reduced at the cost of needing to navigate and search through more pages and a more complex hierarchy; (2) the short hyperlink labels may be enigmatic, particularly for those with SMI; and (3) the reduced information on a page may actually make it harder to understand the information being provided due to the limited context, whereas more information, to an extent, may improve users’ understanding of the contents. More information can have some advantages, but is not independent of quantity or its organization as our results indicate. Providing more information can also necessitate longer pages, which may require scrolling. Scrolling has negative consequences associated with it. Achieving this without scrolling is the ideal solution, but it may be far better to make users with SMI scroll than to navigate to a new page and through a more complex hierarchy [22].

Given the novelty of several of the complexity concepts (eg, navigation areas, topic areas), we assessed variations in ways to measure them to explore whether a particular metric might prove superior for predicting usability. All variations performed similarly using this dataset.

Limitations

The influence on a website’s usability of users’ abilities to understand the text was potentially only grossly measured by assessing a page’s reading level and ease. It might also be that the hyperlink, versus nonhyperlink text, was far more understandable on some sites than others, and that this subcomponent of the text had a greater influence on usability.

The tasks did not assess the ability to understand content but rather the ability to find contents. This has implications for interpreting these findings. For example, pictures and diagrams may inhibit navigational efficiency; however, in the correct context, they may aid in content understanding. We examined the usability of navigational pages, not the effectiveness of content pages.

Given the number of variables to websites, it is important to acknowledge that the results might not be the same with a different set of websites. Also, the conclusions must be tempered because of the collinearity among several variables. Having said this, we have had findings consistent with these in studies that used far more websites [28].

We have conducted similar evaluations to what are presented here on websites that we designed from scratch. The work reported here was intended to evaluate these design principles on actual websites available to the public. The use of real websites means that the range of designs is more limited than if the websites were constructed solely to evaluate design principles. However, this provides evaluation of these design principles on real-world apps, which is important for understanding the significance of design to the usage of actual websites. In this context, it must be pointed out that the FEDM does not necessarily apply to apps where navigation routes are predefined by design, such as in software installation “wizards,” or similar apps where the user does not self-navigate but simply answers questions and the navigation path is preprogrammed based on responses to questions by the designer [25], and thus, in such apps, the hierarchy can be quite deep with many branching points, and still not be complex from a user’s perspective. We only tested the websites on persons who had an SMI. Consequently, we do not know the extent to which these findings are relevant to other individuals.

The participants in this study did not include younger individuals in their teens and twenties. It is possible that these findings will not be as applicable to those who are earlier in their illness, or are more familiar with technology. Preliminary analyses of our data, yet to be published, indicate that age is not as important as expertise with technology in determining which designs are more or less usable.

Conclusions

Websites may contain vast resources; some have millions of pages [34]. Research to make such apps usable by persons with SMI, and others with cognitive impairments, who may also have limited technology experience, is still in its early stages[20,22,23,30]. A recent systematic review found only 10 studies, in addition to 3 we have conducted, which assessed barriers (and potential responses) to website use by individuals with mental illness (broadly defined) [35]. They found 42 barriers and 59 potential responses. Although some specific design responses to identified barriers do exist, and the number of evidence-based responses is growing, for many barriers, only very general guidance can be offered at present. The evidence-based guidance that has been published offers a foundation for creating more usable designs.

Given the growing use of technology-delivered health care services, a major public health challenge is to create design models for those with special cognitive needs and low technology expertise, such as persons with SMI. Our findings, coupled with the findings of others who have shown that eHealth apps designed using standard models are not usable by many with schizophrenia [22], imply that apps designed specifically to accommodate the cognitive needs of persons with SMI are needed. When this is done, these designs can be more usable, and their use is more effective and efficient than those designed using standard models. This study further highlights these conclusions. The FEDM provides a model that can be used to aid the design of eHealth apps. It has been created based on empirical findings from usability studies and developed to provide specific recommendations for creating accessible designs. The FEDM’s 3 dimensions provide practical constructs and guidance. At the simplest level, the 3 dimensions of navigational simplicity, reduced page complexity, and comprehensibility offer help in designing sites that support key tasks that users must accomplish during successful eHealth apps usage. Beyond these simple goals, the research shows the variables that contribute in each dimension allowing designers to estimate the impact of such things as the number of hyperlinks or images on a given page and how they relate to page complexity. Although some research has been conducted to identify design barriers to effective use of eHealth technologies, including our own, there is a clear need for additional research on both barriers and solutions that use rigorous experimental designs to enhance validity.