Published on in Vol 8, No 1 (2021): Jan-Mar

Preprints (earlier versions) of this paper are available at, first published .
Applying Website Rankings to Digital Health Centers in the United States to Assess Public Engagement: Website Usability Study

Applying Website Rankings to Digital Health Centers in the United States to Assess Public Engagement: Website Usability Study

Applying Website Rankings to Digital Health Centers in the United States to Assess Public Engagement: Website Usability Study

Original Paper

1Department of Research, Rocky Vista University College of Osteopathic Medicine, Parker, CO, United States

2Division of Psychology and Language Sciences, University College London, Bloomsbury, United Kingdom

3Center for Innovation in Digital HealthCare, Massachusetts General Hospital, Boston, MA, United States

4Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA, United States

Corresponding Author:

Joshua David Calvano, BS

Department of Research

Rocky Vista University College of Osteopathic Medicine

8401 S Chambers Rd

Parker, CO, 80134

United States

Phone: 1 3033732008


Background: As the public increasingly uses the internet to search for resources and information regarding health and medicine, it is important that health care organizations provide adequate web resources. Website usability refers to the ease of user experience on a website. In this study, we conducted usability analyses on digital health center websites.

Objective: The primary aims of this study were to (1) replicate a preexisting usability scoring methodology for digital health centers; (2) apply and test this replicated usability scoring methodology on a sample set of digital health center websites; and (3) derive recommendations from the results on potential areas of improvements for our sample of digital health center websites.

Methods: Website usability testing was conducted from March 1, 2020, to March 15, 2020. We replicated a methodology and scoring system from previous literature and applied them to digital health center websites. Our sample included 67 digital health centers that were affiliated with US universities or hospital systems. Usability was split into the following four broad categories: accessibility, marketing, content quality, and technology. Usability tools were used to score websites in each of the four categories. The composite of the key factors of each category was used to generate a general usability and overall usability score for each website.

Results: The category with the highest average score (6.3) was content quality. The content quality score also had the highest SD (2.18) and an SE of 0.27. The lowest performing category was technology, which had an average score of 0.9. The technology score also had the smallest SD (0.07) and an SE of 0.01.

Conclusions: Our data suggest that content quality, on average, was the highest scoring variable among digital health center websites. As content is crucial to digital health knowledge, it is justified that digital health centers invest more resources into creating quality content. The overall lowest scoring variable was technology. Potential reasons for this finding include designated funding for servers, a lack of regulatory frameworks for social media presence and liability, and infrequent website audits. An easy approach for improving this variable is increasing website speed. Accessibility is another area that organizations can potentially improve. We recommend that these organizations perform periodic audits of their web presence with usability tools.

JMIR Hum Factors 2021;8(1):e20721




A hospital’s or digital health care center’s website is often these organizations’ first point of contact with the public; therefore, websites are crucial in first impressions [1,2]. They have the potential to be an important part of the first step in improving patient satisfaction and attracting new patients [3]. In a time when information is expected to be readily available, health care organizations use their websites as key tools for both patient communication and education [4-6]. Patients expect to find current and reliable information on websites that are easily accessible in order to make health-related decisions [7]. As many health-related sources are available (eg, WebMD), health care organizations are aiming to improve their internet presence so that they can better communicate with and market to potential customers [3].

Website Usability

Improving website usability is a noteworthy approach that medical organizations can use to improve their internet presence, attract and retain more users, and disseminate accurate and reliable information to a larger audience. Usability goes beyond surface-level design; it broadly refers [8] to a product’s user experience, which includes aspects such as the ease of navigation or user-encountered problems within a website [9]. It addresses the question of how easy or pleasing a website is to use, which are factors that can influence the number of users that engage with a website. Usability also addresses users’ level of engagement and a website’s ability to achieve other objectives. When users are not able to easily access and use a website, they are unlikely to continue using it as an information source. Alternatively, improved usability can enhance the reach of a website. It is for this reason that websites are facing the increasing need to conform to user expectations, desires, and requirements [10,11]. Various industries have established standardized guidelines for accessibility, content, marketing, and technology to improve website usability [12-14].

Usability Studies for Digital Health Centers

Studies have sought to apply usability analyses to e-commerce, e-governments, mobile news apps, and library websites [15-18]. In health care, other studies have analyzed the usability of hospital, children’s hospital, and cancer center websites [3,19,20]. However, to our knowledge, no usability studies have been conducted for digital health centers in the United States. Digital health centers combine innovation-driven health care research with digital technology. Digital health technologies are emerging tools that have the potential to improve patient-centered health care by improving care quality and reducing health care costs [21]. Given digital health centers’ focus on digital technologies (eg, technologies for improving web presence), there is an opportunity to better understand how digital health centers are adapting to technologies that use their web presence. Specifically, we believe that it is distinctly important for these organizations to create websites with high usability to not only improve user experience but also present themselves as leaders in innovation.


The primary aims of this study were (1) to replicate a preexisting usability scoring methodology for digital health centers; (2) to apply and test this replicated usability scoring methodology on a sample set of digital health center websites; and (3) to derive recommendations from the results on potential areas of improvement for our sample of digital health center websites.

Sample Selection

Our focus was on digital health centers that were affiliated with US universities or hospital systems. Indexing the websites of all digital health centers, such as medical companies, was not within the scope of our study.

The original sample set was derived from Becker’s Hospital Review and consisted of a total of 66 digital health centers [22]. We augmented this sample set by including 8 additional digital health centers that were found with Google’s Advanced Search query builder, which increased the total number of digital health centers in our sample to 72. The terms and phrases searched included “academic digital health center,” “academic innovative health center,” “hospital innovation center,” and “hospital digital health,” and the selected region of interest was the United States. We excluded three digital health centers that did not have a designated digital health center website. Our final sample set consisted of 70 digital health centers (Figure 1).

Figure 1. Sample selection criteria for digital health center websites.
View this figure


Website usability tests were conducted from March 1, 2020, to March 15, 2020. The methodology used in this study was replicated from previously published health care usability literature [3,19]. We chose to evaluate the same factors as those in the previous studies. However, we modified the definitions for clarity and reproducibility by using our selected assessment tools (Multimedia Appendix 1). We used the same weighted percentages as those from the previous studies and applied specific formulas to these calculated percentages to create a relative scale for comparing usability scores.

In alignment with the replicated methodology, websites were assessed with four scales for the following categories: (1) accessibility, which refers to the ability of people with low levels of computer literacy to access and navigate hospital’s web presence; (2) marketing, which refers to the ability to be found through search engines and the relevance of descriptions to the links provided; (3) content quality, which refers to grammar, the frequency of information updates, material relevancy, and readability; and (4) technology, which refers to download speed, the quality of the programming code, and website infrastructure [3,19]. Each of these categories represent distinct, quantifiable, and actionable areas of usability that digital health centers can improve on to communicate more effectively with their audiences.


All websites were analyzed by using a set of established usability tools (Multimedia Appendix 1). The tools were chosen based on their ability to meet the industry standards for evaluating the selected factors and their relative ease of use. The process for using each tool was based on the tool’s specific instruction manual. One author of this manuscript supervised the training for a team of 6 student reviewers. Each reviewer underwent the same training for using the suite of analytic tools and performing data entry. The reviewers were then directly observed while they used each tool on three websites, in order to confirm proper usage and reliability. Discrepancies and questions were addressed and answered by the supervising author as they arose. In addition, each tool was used on the same local computer to account for irregularities such as differing internet service providers or computing components, which might affect the consistency and reliability of the results. Factors that can vary from second to second, such as speed, were averaged across two separate tools to provide the most accurate values possible.

We built a database of the top-level URLs that were associated with each website in our data set. This was done by using a web crawler, which is a tool that processes URLs and creates topographical maps of a website and all of its subpages. For instance, a top-level domain that corresponds to a website’s home page may be associated with the URL A subpage of this center’s website might be a page about the team members, which might be associated with the URL There may be other subpages for specific topics, such as the emergency medicine department and the pediatric department. Once the web crawler creates a topographical map of a website, that website can then be analyzed for page errors, the amount of page content, metadata (ie, titles, keywords, and descriptions), or other preprogrammed factors [23].

Websites in our data set received a final score for the following four categories: accessibility, content, marketing, and technology. Per the replicated methodology, the composite of select key factors across each of the four categories was used to generate a fifth general usability score for each website. A weighted aggregate of these five scores was used as the sixth and final score for the final ranking system.

In the following sections, we describe each of the categories that we evaluated, the development of the rating scale for each of these categories, and the importance of each category’s contributions.


The accessibility rating indicates a website’s appeal to a broad audience of people with varying literacy levels, technical aptitudes, and disabilities. This category involves factors such as a web page’s meta-description, readability, and the overall layout of the website. A meta-description is the “snippet” page summary that appears in search results when using a search engine. Another factor is functionality, which encompasses elements like actionable buttons that send users to parts of a website and content that is understandable to people with a wide range of education levels or reading abilities. For instance, it has been reported that an estimated 43% of American adults have basic or below basic literacy levels [24]. Accessibility ratings can also be used to evaluate the usability of assistive technologies, such as screen readers and magnifiers for a given website [25]. For our study, we used the Flesch-Kincaid Reading Ease and Gunning Fog Index algorithmic readability scales to rank a website’s reading difficulty and approximate the grade level required to understand each website’s content.

Content Quality

The content quality rating is used to assess the content on a website. This can include the relevancy of written information to a particular point in time and a specific topic, generated metadata, and the use of a website’s multimedia elements. For instance, a website that is dedicated to supplying information on current closed-loop insulin pumps for people with diabetes may be evaluated on its ability to provide relevant, fact-driven answers to people who seek such information (eg, relative costs, ease of use, etc). In content quality analysis, the multimedia elements on a website may be evaluated for their quality (eg, resolution) and their ability to support the website’s content (ie, available metadata functions). Content quality analysis also involves the assessment of written text (ie, the evaluation of grammar and spelling).


The marketing rating indicates the discoverability of a website. This rating puts particular emphasis on search engine results pages (SERPs), which refer to websites that are suggested to users when they search for information via a web-based search engine, such as Google. Higher placement in search results can lead to greater visibility, and SERPs are considered by some as one of the most important elements of digital marketing. The field of search engine optimization (SEO) involves optimizing a website to achieve higher placement in SERPs, and effectively implementing SEO methods may allow health care organizations to uphold their corporate image as industry leaders [26]. Technical SEO auditing was beyond the scope of this study.


The technology rating indicates the technical functionality of a website as opposed to its content quality; it indicates the quality of a website’s technology, technological design, and performance. The technology rating encompasses various aspects, including front-end design, user experience, back-end coding infrastructure, and server management. The front end is what users view when browsing a website. Front-end design involves analyzing HTML elements to ensure that a user is provided with an easily navigable layout and that the website is scalable across devices (ie, computers, mobile phones, and tablets). The back end refers to the programming code upon which the website runs. This code and other website components, such as databases, are stored on servers, which functionally allow people to view websites from their own devices. The servers also affect the speed of the website (eg, how quickly it loads for users), which can play a crucial role in gaining and maintaining users and followers. For instance, a previous study conducted by Google [27] showed that a website that takes longer than 3 seconds to load on a mobile device will lose approximately 53% of its users. Furthermore, the study revealed that the average mobile website speed is upwards of 18 seconds [27].

General Usability

The general usability rating was based on a composite of select key factors from the prior four categories. The concept of general usability aims to answer the question “how good is my website?” This metric may serve as a starting point for health care organizations to perform an initial audit of their website and identify areas of improvement.

Overall Usability

An overall usability rank order calculation was performed to comprehensively evaluate all major and minor factors across the five aforementioned categories. Afterward, we assigned weighted percentages to all factors to create an all-inclusive usability ranking system.

Technical issues prevented the web crawler from running on three websites. This was possibly due to the fact that no index restrictions were set up by the website administrators. We assigned scores to the remaining (N=67) digital health centers.

The subcategory with the highest average score (6.3) was content quality. The content quality score also had the highest SD (2.18) and an SE of 0.27. Accessibility was the second highest scoring subcategory, which had an average score of 2.2. The accessibility score had an SD of 0.51 and an SE of 0.06. Of the four subcategories, marketing had the third highest average score (1.5). The marketing score had an SD of 0.40 and an SE of 0.05. The lowest performing subcategory was technology, which had an average score of 0.9. The technology score also had the smallest SD (0.07) and an SE of 0.01. The summary statistics for all five categories are presented in Table 1.

Table 1. Digital health center website summary statistics from the usability analysis.
CategoryScore, mean (SE; SD)Score, range
Accessibility2.2 (0.06; 0.51)0.9-3.3
Content quality6.3 (0.27; 2.18)1.1-10.7
Marketing1.5 (0.05; 0.40)0.6-2.4
Technology0.9 (0.01; 0.07)0.7-1.0
General usability1.5 (0.04; 0.33)0.8-2.2

The overall rankings for the 67 assessed domains across all categories are reported in Multimedia Appendix 2. The highest scoring centers across all five usability ranking categories were as follows: (1) Sutter Health Design and Innovation (accessibility score=3.3); (2) Sutter Health Design and Innovation (content quality score=10.7); (3) Mayo Clinic Center for Innovation (marketing score=2.4); (4) University of Texas Southwestern Office for Technology Development (technology score=1); and (5) Sutter Health Design and Innovation (general usability score=2.2). In terms of overall usability, the top performing website was that of Sutter Health Design and Innovation (overall usability score=3).

Comparison With Prior Work

Emerging technologies in the field of digital health are rapidly changing the aspects of health care by making them more patient centered, improving care quality, and decreasing health care costs [21]. The increasing importance of digital health has made it an appropriate field for website usability research.

Our study involved methods that were replicated from previous studies. This allowed us to assess similar trends across various health care website dimensions [3,19]. As with previous studies, the overall scores in our study were highest for the content quality category. This finding could reflect the importance of information to the health care industry and indicate that health centers should invest most heavily in content quality when creating their websites.

Another finding that was consistent with prior research was that the overall lowest ranking category was technology [3,20]. This may be due to a lack of investment in digital technology by the health care industry (eg, investments in server capacity, social media, and website audits). One approach for immediately improving technology is improving website speed. This is largely accomplished through modifying back-end web server settings and minimizing the number of conflicting technologies that run on a website.

A study that evaluated children’s hospitals found accessibility to be the lowest ranking category, which was not the case in our study. However, our accessibility scores were lower than originally anticipated [19]. With regard to health care, we believe that accessibility should be paramount. Health industry leaders should put more effort into ensuring that all domains remain functional and accessible to everyone, so that the quality content on these websites can reach the appropriate users [25].


This study has several limitations. It is common for large organizations to have specific subpages that are dedicated to digital health. For example, one website had an estimated domain age of 33 years when, in reality, the associated innovation health center was aged less than 5 years. However, structuring websites in this manner may provide digital health centers with a competitive advantage, as this method would improve their rankings. This would result in an increase in the number of people who view their website.

Additional limitations and difficulties were found in the assessment of social media websites. Not all social media accounts were directly accessible from these websites. As such, it was difficult to find certain social media accounts through Twitter’s and Facebook’s respective search engines. Oftentimes, the digital health centers’ profiles were distant from the top result.

Assessments of a website’s speed can vary depending on the time of day or the day of data collection. This could be due to changes in the website’s servers, internet connectivity, or computer hardware. To minimize sampling bias, the same computer and the same network were used for all of our tools.

Another limitation was that all information was collected over the course of 2 weeks. As such, several measures may have changed since the initial evaluation.


With digital health emerging as a leading field in terms of innovation in health care, it is important that digital health centers are able to effectively connect with the public by using their websites. In this study, we conducted an analysis of the overall need for improving the usability of digital health centers’ websites. The average general usability score was 1.5. This shows the necessity of improving the usability aspects of websites. Digital health centers may benefit from taking steps to improve the various components of their websites in order to reach their audiences. A suggested step for these organizations is to perform periodic usability audits of their websites to identify areas for improvement. Several of these institutions have considerable room for improvement in terms of their overall web presence. We have identified approaches that these organizations can use to increase their websites’ usability, such as improving website speed and social media access. These approaches could potentially improve their websites’ reach.


We would like to thank the following individuals from the Rocky Vista University College of Medicine for their contributions in data entry and ongoing support: Brandon Bealer, Kameron Black, Joseph Fike, Andrew Glerum, and Ryan Henschell.

Conflicts of Interest

SS and SH helped to create the website for the Massachusetts General Hospital Center for Innovation in Digital HealthCare. The website was excluded from this study.

Multimedia Appendix 1

Defined usability factors with their associated percentage weights, assessment tools, impacts, and formulas.

PDF File (Adobe PDF File), 36 KB

Multimedia Appendix 2

Digital health center websites and category scores.

PDF File (Adobe PDF File), 91 KB

  1. Alpay LL, Overberg RI, Zwetsloot-Schonk B. Empowering citizens in assessing health related websites: a driving factor for healthcare governance. International Journal of Healthcare Technology and Management 2007;8(1/2):141. [CrossRef]
  2. Revere L, Robinson L. How healthcare organizations use the Internet to market quality achievements. J Healthc Manag 2010;55(1):39-49; discussion 49. [Medline]
  3. Huerta TR, Hefner JL, Ford EW, McAlearney AS, Menachemi N. Hospital website rankings in the United States: expanding benchmarks and standards for effective consumer engagement. J Med Internet Res 2014 Feb 25;16(2):e64 [FREE Full text] [CrossRef] [Medline]
  4. Coile RC. E-health: reinventing healthcare in the information age. J Healthc Manag 2000;45(3):206-210. [Medline]
  5. Randeree E, Rao HR. E-health and assurance: curing hospital websites. Int J Electron Healthc 2004;1(1):33-46. [CrossRef] [Medline]
  6. Wen K, Kreps G, Zhu F, Miller S. Consumers' perceptions about and use of the internet for personal health records and health information exchange: analysis of the 2007 Health Information National Trends Survey. J Med Internet Res 2010 Dec 18;12(4):e73 [FREE Full text] [CrossRef] [Medline]
  7. Reid P, Borycki EM. Emergence of a new consumer health informatics framework: introducing the healthcare organization. Stud Health Technol Inform 2011;164:353-357. [Medline]
  8. Griffiths KM, Christensen H. Website quality indicators for consumers. J Med Internet Res 2005 Nov 15;7(5):e55 [FREE Full text] [CrossRef] [Medline]
  9. Otis M, Zhu J, Mustafa-Kutana SN, Bernier AV, Shum JM, Dupre JAS, et al. Testing usability and feasibility of a mobile educator tool for pediatric diabetes self-management: Mixed methods pilot study. JMIR Form Res 2020 May 01;4(5):e16262 [FREE Full text] [CrossRef] [Medline]
  10. Mittler JN, Volmar KM, Shaw BW, Christianson JB, Scanlon DP. Using websites to engage consumers in managing their health and healthcare. Am J Manag Care 2012 Sep;18(6 Suppl):s177-s184. [Medline]
  11. Liang C, Chen H. A study of the impacts of website quality on customer relationship performance. Total Quality Management & Business Excellence 2009 Sep;20(9):971-988. [CrossRef]
  12. Oermann MH, Lowery NF, Thornley J. Evaluation of Web sites on management of pain in children. Pain Manag Nurs 2003 Sep;4(3):99-105. [CrossRef] [Medline]
  13. Oermann MH, Lesley ML, VanderWal JS. Using Web sites on quality health care for teaching consumers in public libraries. Qual Manag Health Care 2005;14(3):188-195. [CrossRef] [Medline]
  14. Oermann MH, McInerney SM. An evaluation of sepsis Web sites for patient and family education. Plast Surg Nurs 2007;27(4):192-196. [CrossRef] [Medline]
  15. Kuan H, Bock G, Vathanophas V. Comparing the effects of usability on customer conversion and retention at e-commerce websites. : IEEE Computer Society; 2005 Presented at: Proc Annu Hawaii Int Conf Syst Sci; January 6, 2005; Big Island p. 174   URL: [CrossRef]
  16. Huang Z, Benyoucef M. Usability and credibility of e-government websites. Gov Inf Q 2014 Oct;31(4):584-595. [CrossRef]
  17. Jeong W, Jung Han H. Usability study on newspaper mobile websites. OCLC Systems & Services 2012 Oct 26;28(4):180-198. [CrossRef]
  18. N VR, Harinarayana NS. An analysis of usability features of library web sites. Annals of Library and Information Studies 2008 Jun;55(2):111-122 [FREE Full text]
  19. Huerta TR, Walker DM, Ford EW. An evaluation and ranking of children's hospital websites in the United States. J Med Internet Res 2016 Aug 22;18(8):e228 [FREE Full text] [CrossRef] [Medline]
  20. Huerta TR, Walker DM, Ford EW. Cancer center website rankings in the USA: Expanding benchmarks and standards for effective public outreach and education. J Cancer Educ 2017 Jun;32(2):364-373. [CrossRef] [Medline]
  21. DePasse JW, Chen CE, Sawyer A, Jethwani K, Sim I. Academic medical centers as digital health catalysts. Healthc (Amst) 2014 Sep;2(3):173-176. [CrossRef] [Medline]
  22. Dyrda L. 66 hospitals and health systems with innovation programs. Becker's Hospital Review.   URL: https:/​/www.​​lists/​66-hospitals-and-health-systems-with-innovation-programs-2018.​html [accessed 2020-05-02]
  23. Devi RS, Manjula D, Siddharth RK. An efficient approach for web indexing of big data through hyperlinks in web crawling. ScientificWorldJournal 2015;2015:739286 [FREE Full text] [CrossRef] [Medline]
  24. Kutner M, Greenberg E, Jin Y, Boyle B, Hsu Y, Dunleavy E. Literacy in everyday life: Results from the 2003 National Assessment of Adult Literacy. National Center for Education Statistics. 2007.   URL: [accessed 2021-03-25]
  25. Ismail A, Kuppusamy KS, Nengroo AS. Multi-tool accessibility assessment of government department websites:a case-study with JKGAD. Disabil Rehabil Assist Technol 2018 Aug;13(6):504-516. [CrossRef] [Medline]
  26. Batraga A, Rutitis D. Corporate identity within the health care industry. Economics and Management 2012 Dec 07;17(4):1551. [CrossRef]
  27. New Industry Benchmarks for Mobile Page Speed. Think with Google.   URL: https:/​/www.​​marketing-resources/​data-measurement/​mobile-page-speed-new-industry-benchmarks/​ [accessed 2020-05-01]

SEO: search engine optimization
SERP: search engine results page

Edited by G Eysenbach; submitted 26.05.20; peer-reviewed by M Nitsch; comments to author 22.07.20; revised version received 02.08.20; accepted 02.03.21; published 29.03.21


©Joshua David Calvano, Edwin Lauritz Fundingsland Jr, Deborah Lai, Sara Silacci, Ali S Raja, Shuhan He. Originally published in JMIR Human Factors (, 29.03.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.