SARS-CoV-2, a novel coronavirus, was first detected in the United States in mid-January 2020 [1,2], and eventually, many states enacted stay-at-home orders in early March. The SARS-CoV-2 pandemic challenged the public health system in the United States in many ways, including a lack of laboratory testing capacity early in the pandemic, evolving data standards for reporting positive test results and deaths owing to COVID-19, and a lack of coordination among state and federal agencies. In addition, approximately half of all nonfederal hospitals lacked the capacity to electronically exchange information with public health agencies at the beginning of the pandemic [3].

The pandemic also presented challenges in communicating with the public the evolving status of the outbreak and the reasoning behind public health measures, such as stay-at-home orders and masking. As the pandemic progressed, waves of infection rose and fell in the regions of the United States at different times owing to the presence of superspreader events, differences in public health responses among states, and the rise of variants [4,5].

Data visualization has the potential to modify the course of a pandemic by bringing together information about the state of the pandemic, public policy, and individual behavior in ways that are actionable. However, for the visualizations to have this impact, they must be easily accessible, based on accurate and timely data, and carefully developed with an understanding of both the data and the principles of visual design.

Visualizations have been easily accessible during the pandemic owing to the availability of numerous software tools and platforms for creating graphics and mapping data. Because data sets are available to anyone with an internet connection, early in the pandemic, a number of visualization experts wrote about the need to responsibly use tools and data when creating visualizations [6-9]. Misrepresented and misinterpreted COVID-19 visualizations have inspired one study to use them to help students develop statistical literacy [10].

The large number of visualizations developed and deployed rapidly by public health authorities and data analysts during the pandemic is of interest to visualization and communication researchers. They provide insights and lessons about the process of rapidly designing and developing visualizations [11-15]; efforts to curate global data [16]; the types of visualizations created and who they are for [17-19]; conceptual models linking tools, data, visualizations, and users [20]; and what it means for a visualization to be actionable [21]. In addition, studies have used pandemic data to understand how the users perceive the risk and severity of the pandemic [22-24] and their reactions to the designs of dashboards [25].

This study complements earlier work by taking a US-focused look at COVID-19 dashboards from August 2020 to June 2022. It documents data sources and data aggregation efforts, identifies themes relevant to designing dashboards for outbreaks, and highlights issues with data availability and standardization. The goals of this work were to provide insights that will help application developers increase the usefulness, transparency, and trustworthiness of dashboards and trackers for public health data in the future and to document the variety of dashboards and trackers used by the residents of the United States and the evolution of these tools for approximately 2 years.

This study encompassed the following 2 broad categories of data visualization: dashboards and trackers. The term dashboard generally refers to a set of dynamically updated data visualizations placed in proximity to one another and is used to monitor conditions for the purpose of understanding a system or event. Because several COVID-19 data visualizations took other forms, such as visualizations arranged sequentially with accompanying text, I used the term tracker to more broadly refer to these types of dynamically updated displays.

To identify web-based dashboards and trackers, I performed a web search using Google on August 12, 2020. Searches were formatted as “coronavirus COVID dashboard tracker” combined with either a state name, “United States,” or “global.” The first 15 results for each keyword combination were examined for their relevance. On the basis of test searches, I determined that the relevant search results were generally in the first 10 results, and results ranked lower than the 15th search result were either links to the dashboards and trackers from other web pages or news or commentary about the pandemic. The websites that were determined to be relevant to this study are listed in Multimedia Appendix 1. Dashboards and trackers were categorized as state focused, nationwide, or global. This study focused on visualizations that are the most likely to be viewed by people in the United States to understand the local and regional status of the pandemic, with less emphasis on global visualizations.

By the end of January 2021, many states had incorporated vaccine dashboards into their state dashboards. To locate vaccine dashboards not integrated into state websites, I performed a Google search for “covid vaccine dashboard tracker” and examined the first 20 results for relevance. Multimedia Appendix 2 lists the websites determined to be relevant to this study.

Websites were included in this survey if they displayed up-to-date information, appeared to be updated daily (or nearly daily), and relied mainly on graphs or maps (rather than tables or text) to convey information. Websites were excluded if they showed data limited to regions smaller than a state (such as a single county or city), were specific to a type of setting (such as prisons), or displayed only trackers or dashboards that had been embedded from other websites. The District of Columbia was included in this survey, but the US territories were not. This survey included only publicly available websites accessible on a laptop. Apps developed specifically for smartphones were not included.

The focus of this study was the display of information concerning diagnosed cases of COVID-19, deaths attributed to COVID-19, testing for COVID-19, and vaccination. Visualizations of risk levels, hospital bed availability, and hospital admissions were not central to this study, but designs for these types of data may benefit from these findings.

All websites were examined on a MacBook Pro laptop (Mac OS version 10.11) using a Firefox web browser. The data sources used by each dashboard or tracker were documented based on statements from the website itself [26], and for one dashboard, the development team was contacted. Some websites included a statement stating that their data sources have changed as the pandemic has developed, suggesting that their list of sources may not be complete or current. The software tool or method used to create visualizations was also determined. If the name of the software brand was not displayed with the visualization, the Inspector tool within Firefox was used to examine the webpage’s HTML and determine the tool or method used.

Each website was examined to determine the following:

To understand how the dashboards and trackers evolved over the course of the pandemic, the survey of websites was repeated 3 more times. This survey spanned from approximately 7 months after the novel coronavirus was first detected in the United States to nearly 2-and-a-half years after detection. The second review of each website was conducted between January 2021 and March 2021. The third review of each website was conducted in either December 2021 or January 2022. The final review was conducted in June 2022. By the end of the survey period, >1 million deaths in the United States were attributed to COVID-19, with deaths decreasing to <400 per day by June 2022.

Multimedia Appendices 1 and 2 list the information for the websites and any changes to the URLs are noted in the appendices. This review covers only the dashboards and trackers identified in August 2020 and vaccine-focused websites identified from January 2021. Therefore, websites launched after that time were not included.

On the basis of a review of the websites, 5 sets of themes were developed to highlight issues concerning challenges in presenting COVID-19 data and techniques of effective visualization.

The most popular software platforms used for state-focused dashboards and trackers, particularly among public health authorities, were Tableau, ArcGIS, and Microsoft Power BI. Some dashboards presented all the information in a single page, but it was common for dashboards to have multiple pages to accommodate maps and new types of data that became available during the pandemic. News organizations were more likely to provide trackers arranged as a series of data visualizations with textual explanations and use scalable vector graphics embedded in their web pages. See Multimedia Appendix 1 for information on the visualization tools or methods used for each dashboard and tracker.

Trust and transparency are emphasized in the guidelines the World Health Organization has assembled for communicating with the public about disease outbreaks [34]. Dashboards and trackers may inform viewers of visualizations about the sources of the data in several ways. The most direct approach is to provide the data source within a caption for each map or graph; however, this may not be feasible for dashboards combining several visualizations. Websites using data aggregators often simply state one or more sources for the entire collection of visualizations. One nationwide dashboard [N-11] was particularly vague about the relationship between the visualizations displayed and the sources of data, crediting CDC, WHO, The New York Times, JHU, Corona Data Scraper, and official state and county health agencies without providing further details. When websites list sources in this manner, it raises the following questions:

Early in the pandemic, data scientists raised concerns about the quality of COVID-19 data [35,36]. The challenges of collecting global data appropriate for display and analysis have led to questions regarding the methodologies and sources used by some aggregators. For example, Worldometer is a private company known for its web counters that estimate world statistics. It became an aggregator of COVID-19 data and provider of popular COVID-19 trackers [G-12] and has been criticized for having an anonymous curation team and opaque methodology [37].

Multimedia Appendix 1 presents the tools and methods used to construct the visualizations examined in this survey. With the exception of Massachusetts, all state public health authorities provided a web page displaying a dashboard or tracker in August 2020 (with Massachusetts providing PDF downloads). Websites of state public health authorities were often constructed using ArcGIS, whereas state-focused websites from other types of developers relied on a variety of tools (including Tableau, Datawrapper, Infogram, and Microsoft Power BI). Websites providing nationwide coverage tended to be constructed with frameworks using embedded scalable vector graphics. Global dashboards and trackers were created with a variety of methods.

This study identified and examined >100 websites providing COVID-19 dashboards and trackers relevant to the residents of the United States and highlighted the multitude of factors that affect these visualizations. The findings reveal the role data aggregators have played in making data accessible to visualization developers as well as lapses in communicating to viewers the provenance of the data. Decisions by public health experts about data collection and data standards have downstream effects on which data are available to be communicated and compared. In addition, each step of this process is impacted by the evolving nature of the pandemic and political and social systems.

The five themes identified in this work can guide future development of visualizations of public health data for the public: (1) viewers should be made aware that data are an imperfect representation of reality owing to methods of data collection and reporting; (2) viewers need context for interpreting visualizations, such as comparisons with other data or indicators of relevant events on timelines; (3) developers should carefully consider whether plotting a raw data stream, cumulative values, or transformation of values will be the most useful to viewers; (4) the graphical form of a visualization should be chosen to fit the type of data and be designed to make good use of perceptual principles; and (5) visualizations designed to use automated streams of data must be monitored to ensure that the data continue to have reasonable values and that the design of the visualization remains useful with the new data.

One of the persistent challenges faced by data aggregators has been managing disparate data sets for analysis and visualization. In the United States, the collection of public health data is governed at the local and state levels [50]. Strategies differ by state, with no central government authority to standardize data collection and reporting. The Council of State and Territorial Epidemiologists published standards for the clinical diagnosis of COVID-19 and data elements to report in April 2020, with updates in August 2020 and August 2021 [40,45,51]. The Council of State and Territorial Epidemiologists also recommended that states enact laws to make cases of COVID-19 reportable to public health authorities. The CDC has no authority to require reporting, stating “COVID-19 case surveillance data are collected by jurisdictions and reported voluntarily to CDC” [52].

Problems with data quality, standards, and availability have been described by dashboard and aggregator teams [53-56] and journalists [57-60]. Problems in data standardization and availability were somewhat alleviated during the first year of the pandemic, but data on case counts became unreliable by early 2022 because of the introduction of rapid at-home test kits [61,62].

Data that are visualized by a person or an organization that did not originally collect the data is an example of data reuse. The movement around Findable, Accessible, Interoperable, Reusable (FAIR) data includes the responsibility of providing appropriate data citations so that the original source and providence of data are discoverable [63]. The disconnect between the vision of the FAIR data and the findings of this survey is important. One challenge is that COVID-19 data are obtained in frequent updates (rather than from archived data sets) and often from data aggregators. This highlights the gap between the real-world need for trustworthy display of data in public health and typical use cases for using FAIR principles.

What are the purposes of public-facing visualizations of pandemic data, and what data are needed to achieve those purposes?

Dashboards are often described as tools to support decision-making. Visualizations have played an important role in educating citizens about the pandemic and therefore may encourage changes in behavior to mitigate transmission. However, visualizations are likely to have a constellation of purposes. For example, a visualization could help establish trust between public health authorities and citizens. Further, effectively promoting behavior change may depend on first conveying the magnitude of human suffering caused by the pandemic.

The question of what data are useful for decision-making was addressed early in the pandemic by former CDC Director Dr Tom Frieden. He argued that there is a mismatch between the most commonly available data—counts of cases, hospitalizations, and deaths—and the data that are the most useful for guiding COVID-19 response in communities. He suggested that local decision-making for formulating policies should use data that include the number of unlinked infections, number of health care worker infections, and trends in excess mortality [64].

Data visualizations are often assumed to be neutral and objective mechanisms of communication, but they are not. Designing and developing visualizations require numerous decisions regarding the selection of data and methods of presentation. It has been argued that all visualizations are rhetorical and therefore have the power to influence beliefs and behaviors [65,66].

In the context of the COVID-19 pandemic, public health authorities and government officials have made decisions about what data to collect and what data to not collect. These decisions constrain the messages that visualizations can send. In addition, the messages from these visualizations may imply a sense of authority and certainty through their association with organizations that have traditionally been respected (public health agencies, universities, and news organizations) and the “clean lines and structured layouts of traditional visualizations” [65]. This authority and certainty may obscure the extent of human suffering caused by COVID-19, echoing concerns raised by Dragga and Voss [67] in their analysis of graphs depicting fatalities and injuries from causes such as industrial workplaces and baby walkers.

In the United States, the authority of COVID-19 visualizations and the data behind them have been questioned, with various groups asserting that the severity of the pandemic has been either overplayed or downplayed. As many state and local policies for reopening schools and businesses were commonly tied to metrics about the pandemic, such as the test positivity rate or hospitalization rate, people tired of pandemic restrictions have accused COVID-19 data and dashboards of becoming political tools to prevent a return to normal. Other groups adopted a different perspective. For example, in April 2022, a coalition of public health practitioners, scientists, health care workers, educators, and advocates known as The People’s CDC released a statement criticizing the new definitions for categories of community transmission rates. They wrote the following :

Public-facing visualizations of pandemic data are useful only if viewers are able to understand and interpret the data displays they see. Dashboard designers might choose to display large amounts of data with the goal of allowing the viewers to come to their own interpretation of the data without the prescriptive guidance of dashboard designers. However, this effort at transparency can backfire if the viewers are overwhelmed by the complexity or arrive at incorrect conclusions [25,65]. Viewers may assume that websites with more data are more accurate, but the volume of data and visualizations may obscure uncertainties in the data.

Visualization and communication researchers play crucial roles in determining how to better design public-facing dashboards for infectious disease data. Several studies have used COVID-19 data and dashboards in user studies [23-25,69]. Identifying best practices will accelerate the development of effective dashboards and trackers, and the software tools commonly used by public health authorities could incorporate those recommendations into templates. An important area for future investigation is determining if effective design practices for COVID-19 data can be applied to display other types of public health data.

Current research in the field of visualization seeks to develop software tools to assist nonexpert users in choosing effective visualization techniques to support their specific data sets and goals (as demonstrated in studies by Lavalle and Mate [70] and Golfarelli and Lizzi [71]). This aligns with 2 of the themes from this study, choosing the values to display and choosing the graphical form of the visualization. These studies are often based in the domain of business analytics; however, future work could focus on the domain of public health.

This study was limited to dashboards and trackers available to the public as of August 2020 and therefore does not include dashboards used internally by health care and public health organizations. It excludes visualizations produced exclusively for smartphone apps and visualizations that focus on specific populations, such as nursing homes or prisons, or nontraditional data types, such as wastewater sampling.

This analysis reveals the extent to which dashboards and trackers informing the American public about the COVID-19 pandemic relied on an ad hoc pipeline of data sources and data aggregators. The pandemic has been characterized by disparate and evolving data standards, which has complicated the development of dashboards and trackers that display data over time and across regions. The 128 websites of dashboards and trackers identified in this survey offer an opportunity to compare different approaches to the display of similar data. This work highlights examples that provide clarity in interpreting data, and those that obscure the meaning of the data and may potentially mislead viewers.