In this study, we used mixed methods. Participants in this study used VIADS for the hypothesis generation process. For this study, we modified the System Usability Scale (SUS; Multimedia Appendix 1) survey to assess the usability of VIADS. Brooke first proposed the SUS [20,21] in 1996, and it has been widely used to assess the usability of information systems for decades [22-25]. We modified the SUS by including open-ended questions that elaborate and clarify the Likert scale options. For example, if a user selected “disagree” or “strongly disagree” in response to the statement “I think VIADS is easy to use,” a follow-up question asked, “Can you please give an example of how VIADS is not easy to use?” This provided more specific feedback and determined why responses to specific items were unfavorable. The primary objective of this evaluation was to identify improvement opportunities for VIADS. Without explaining the respondent’s score selection, the SUS scores, in our opinion, lacked significant meaning. After the SUS evaluation, VIADS could be enhanced if some negative feedback could be addressed. As a result, we modified the standard SUS (ie, the follow-up questions can assist us in identifying areas that require improvement). Only negative responses were accompanied by a request for clarification.

We administered a 6-question follow-up survey at the end of the study to verify the VIADS usage experience with possible responses of “yes,” “maybe,” “no,” and ”Please elaborate on your answers” (open-ended, optional). Of the 6 questions, 1 question pertained to the overall usefulness of VIADS in clinical research, while the remaining 5 pertained to the specific ways in which VIADS could contribute to the research process. These questions focused on their perception of capacity to (1) provide novel perspectives, (2) facilitate data presentation, (3) facilitate results interpretation, (4) facilitate decision-making, and (5) facilitate other aspects of research. These questions are primarily aligned with the VIADS functionality, with the VIADS design objectives. These are subjective VIADS utility measurements; however, the answers are based on their 1-hour training and 2-hour intense use of VIADS. The objective measures of the utility of VIADS, such as a comparison of the quality of hypotheses generated via VIADS and without VIADS, are currently ongoing and will be shared with readers in separate manuscripts. The cognitive process analysis of the recorded think-aloud sessions is ongoing and will be published separately.

This usability evaluation study was conducted while the participants implemented the think-aloud technique with identical data sets to generate data-driven hypotheses using VIADS. All participants in the study adhered to the same protocol (Multimedia Appendix 2). Multimedia Appendix 3 contains the data extracted from the National Ambulatory Medical Care Survey (NAMCS) conducted by the Centers for Disease Control and Prevention [26,27]. We used data collected in 2005 and 2015 and preprocessed the NAMCS data sets by calculating and aggregating the ICD-9-CM diagnostic and procedural codes and their frequencies. VIADS accepts files in CSV format with 2 columns, one containing ICD-9 codes and the other containing the aggregated ICD-9 code frequencies. The same researcher conducted each study session remotely (via WebEx video conference).

Each participant had a 1-hour training session (Multimedia Appendix 4 contains the training slides that outline the primary functionalities and algorithms of VIADS) followed by a 2-hour study session. In each study session, a participant used the same data sets to perform the analysis; based on his or her experience and knowledge as well as the analysis results, hypotheses were generated, were recorded, and are currently being evaluated by an expert panel. An example of data analysis would be to examine the most frequently used ICD-9 codes in 1 year (2005 or 2015) or to compare the change in ICD-9 code frequencies between 2005 and 2015. During the study session, however, no particular algorithms were requested; each participant was free to explore any algorithms they desired. During the training sessions, the most commonly used scenarios of VIADS were demonstrated to each participant by the researcher. The results reported in this manuscript are based on the participants’ evaluations after the study sessions, which were recorded using BB FlashBack [28] to capture screen activities and conversations between each participant and the researcher. A professional transcription service subsequently transcribed the audio recordings. The modified SUS and an additional follow-up survey containing the 6 questions were administered after each study session. Participants were compensated based on their time spent on the study. Multimedia Appendix 5 is a VIADS user manual with additional information on how to use VIADS specifically.

The data-driven hypothesis generation process results are currently being encoded and analyzed. Once this step is complete, the results will be made public. Therefore, the quality of the hypotheses and the actual cognitive processes involved in hypothesis generation during each study session will be published separately.

The institutional review boards of Clemson University (IRB2020-056) and Ohio University (18-X-192) approved the study. All consent forms and study scripts were shared with all participants prior to the study sessions. The study data sets were shared with each participant on the day of the study session. Verbal permissions were obtained before the study sessions were recorded with each participant.

VIADS was tested by 12 participants, all clinical researchers. They were recruited through multiple national platforms, such as the American Medical Informatics Association discussion forums. Therefore, they were from geographically diverse institutions. Table 1 shows the demographic characteristics of the study participants.

Table 2 shows the SUS scores for each participant. Among the 12 participants, 2 had SUS scores <60, and 5 had SUS scores ≥80. The scores ranged from 37.5 to 87.5. The overall mean SUS score for VIADS was 71.88 (SD 14.62), and the overall median SUS score was 75.

Table 3 presents the detailed raw SUS evaluation results for VIADS without SUS calculations. It summarizes the raw evaluation scores for each SUS evaluation item, with the following range of scores: strongly disagree=1 to strongly agree=5. For one-half of the questions in SUS, higher scores denoted more positive responses (direct questions); for the other one-half, lower scores indicated more positive responses (reverse questions).

The mean results for the direct questions ranged from 3.75 to 4.25 out of 5. The median score for all direct questions was 4. The scores for the reverse questions ranged from 1.92 to 2.83. For the reverse questions, 4 median scores were 2, and 1 median score was 3.

The modified SUS questionnaire and utility questions were asked and answered after a 1-hour training session and a 2-hour study session; when matched to the SUS scores, their answers corroborated their positive usage experience of VIADS. Table 4 presents the results of our VIADS utility questions. As indicated in Table 4, all results were separated into 3 categories: “Yes,” “Maybe,” or “No.” Among the respondents, 100% (12/12) agreed (ie, they all selected Yes) that VIADS provides new perspectives on the underlying data sets, 92% (11/12) felt that it could facilitate the presentation of data sets, and 75% (8/12) agreed that VIADS is a valuable tool for clinical research. Additionally, 75% (8/12) agreed that VIADS could facilitate the interpretation of results and decision-making in hypothesis generation. More than one-half (7/12, 58%) of the participants expressed conservative attitudes when asked if VIADS could assist with other aspects of research (ie, 58% selected either “maybe” or “no” as answers). In addition to subjective measures of the utility of VIADS, we published some objective measures at a conference [29]. For example, participants could generate 5 to 21 hypotheses within 2 hours, and the VIADS group took a shorter time, on average, to generate each hypothesis when we did not consider the quality of the hypotheses. More objective measures (such as the quality of the hypotheses) are still under analysis.

Specific comments on answers to open-ended questions were organized as positive comments and suggestions, some of which were not positive. All positive comments were categorized under thematic headings, and only up to 3 items were presented in Table 5. The themes emerged after we aggregated and synthesized all comments from participants.

The following insights for the improvement of VIADS were answers to the open-ended questions included in the modified SUS: (1) label data sets during comparison and carry the data set labels across pages, (2) more tips to explain the settings while uploading the data sets, (3) include the definitions of the terms and parameters used in VIADS, (4) the data sets accepted by VIADS are very specific, (5) provide further elaboration on the error messages, (6) provide a more detailed description of the functions.

Previous research indicates that the mean SUS usability score is 68, on average, regardless of specific applications (eg, information systems or apps) [22]. The mean SUS score for VIADS in this study was 71.88, and the median score was 75. The literature shows that these are good usability scores [23,30]. Although the average score for VIADS can be improved further, it should be noted that VIADS is a complex analytic tool with many functionalities. The SUS score was encouraging, given the complexity of VIADS and participants' heterogeneous backgrounds. Only 2 of the 12 participants had SUS scores <60. The rest had scores ≥65, and 5 had SUS scores ≥80. Table 2 includes the SUS score for each participant in the VIADS group. Furthermore, the additional questions and constructive insights to improve the VIADS interface and instructions will help us to address these concerns more explicitly.

The average SUS score was 71.88, with an SD of 14.62, which is approximately 20% of the mean SUS. This large SD indicates heterogeneous opinions among participants about the usability of VIADS, allowing us to make more prudent and selective decisions about revisions to VIADS rather than implementing all suggestions. It is possible to investigate the variables contributing to such heterogeneity in a larger sample.

The feedback on the utility of VIADS was predominantly and consistently positive. The follow-up survey results provided some degree of the utility of VIADS, especially after 1 hour of training and 2 hours of using VIADS to analyze the data and generate hypotheses. As a secondary data analytic tool, VIADS fulfills its design purposes. All participants agreed that VIADS offers new perspectives and measures of data sets. The usefulness of VIADS in facilitating data presentation (11/12, 92%), results interpretation, and decision-making in hypothesis generation was agreed upon by at least 75% of the participants. There appeared to be some reservations among the participants about making positive statements on additional aspects of clinical research beyond the dimensions about which they were explicitly asked. However, this could suggest that participants were only prepared to respond to items about which they felt sure. Therefore, we could take these results as additional validation of the positive nature of the overall results, acknowledging that there is always room for improvement.

Among all suggestions to improve VIADS among participants, suggestions 1 (ie, label data sets during comparison and carry the data set labels across pages), 2 (ie, more tips to explain the settings while uploading the data sets), and 5 (ie, provide further elaboration on the error messages) can be added to the VIADS interface. Suggestions 3 (ie, include the definitions of the terms and parameters used in VIADS) and 6 (ie, provide a more detailed description of the functions) are provided in the VIADS user manual and may be highlighted. There is also a legend key in the main interface. Furthermore, point 4 (ie, the data sets accepted by VIADS are very specific) is a limitation of VIADS; although the revisions are ongoing for all other points, point 4 has been excluded. To address point 4, a new tool is needed, which is under development.

Most of the participants positively commented on specific aspects of VIADS. However, it is possible that participants who provided lower SUS ratings were less inclined to leave comments on specific features.

In a system such as VIADS, it can be challenging to balance usability and utility. The functionality of the tool is not simple, and users must understand the underlying algorithms and how to use the tool’s various features and interpret the results it generates. The terms used in the interface alone (eg, NC is for node count, and CC is for the class count) represent a long list of definitions for users to grasp (Figure 1). The comparison summary of VIADS is presented using a single visualization (ie, the ICD-9 hierarchical structure), with highlighted ICD-9 codes if they are statistically different between the 2 data sets. During the development of VIADS, we devoted considerably more time to the utility of the tool, in terms of implementing the desired functionalities, than to the interface’s usability. Although we are encouraged by the SUS scores and the participants’ acclaim for VIADS’s primary features, which is their perception after their intense use of VIADS (ie, 1-hour training and 2 hours of use), actual performance measures are needed and ongoing.

Think-aloud protocols have been used as a method in the evaluation of information systems for decades. Some studies have focused on the investigations of the medical reasoning process [31-35], evaluation of clinical decision support systems [30,36,37], and additional purposes [25,38,39]. Our study used a think-aloud protocol to access the researchers’ thoughts, while participants used VIADS to assess its usability and utility.

We asked the general research question: “Can secondary data analytic tools, such as VIADS, facilitate the hypothesis generation process?” One aspect of the tool related to this question is its usability. Thus, our objective was to investigate the tool’s usability and utility using mixed methods. The process of generating hypotheses using the same data sets via VIADS for clinical research projects was used as a task by participants, which provided real-use experience before participants answered the SUS and utility surveys. The results show the tool’s usability and some degree of utility. VIADS can be constantly updated with users’ feedback. This is an important first step to exploring the role of VIADS in facilitating clinical researchers to generate research and scientific hypotheses and support them at various levels of research.

Furthermore, this useful and accessible tool is freely available online as a user-friendly version, allowing users to leverage the tool without investing unnecessary time in technical details. Our research established a link between using a secondary data analysis tool and facilitating scientific hypothesis generation. This can be a starting point for utilizing secondary data analysis tools to understand the cognitive process of scientific hypothesis generation better.

The study included 12 participants, above the average range for a usability study. Past studies showed that 5 [38], 7 [30], 8 [36], and 12 [37] participants participated in comparable usability studies. The literature indicates that 5 participants can identify approximately 55% of usability issues, while 10 can identify approximately 80% [40]. With 12 participants, we are relatively confident that our usability study has a sufficient number of participants. In addition, our participants were selected from different regions of the country, with varying backgrounds within the clinical research context, providing a more comprehensive perspective of the tool.

Our SUS modification allowed participants to elaborate on the scores assigned to each SUS item. This allowed for targeted VIADS revisions. We believe that our modifications to the SUS were valuable and beneficial additions to the original SUS survey. Despite being grounded in the actual functionality of VIADS, the 6 utility questions and the SUS questions aligned well with the Health Information Technology Usability Evaluation Scale (Health-ITUES) [41]. In terms of health technology assessment frameworks [42,43], VIADS more closely resembles a data analysis tool than a mobile health application. Therefore, the economic evaluation of the tool’s impact deviates slightly from the tool’s primary purpose.

However, we know VIADS accepts only very specific types of data sets, not all. Consequently, the conclusions drawn from the data sets are specific rather than general. Now, we are developing a more generic supporting tool with a broader range of support for researchers.

Question 5 in Table 4 has the lowest agreeable rate; only 42% (5/12) of participants selected “Yes,” and 50% (6/12) selected “Maybe.” This question was supposed to capture any unintended impact of VIADS in addition to the 4 intended functionalities (ie, questions 1 to 4 in Table 4). However, the current presentation of the question can be confusing, which may lead to a low agreeable rate.

We recognize that our usability testing tool (SUS) captures the users’ perceptions, not how VIADS was used. Even though each participant had an intense VIADS use session before they completed the SUS survey, this still is a limitation of this study.

Due to lack of expertise, the graphs generated by VIADS consider more of the meanings and align with the underlying algorithms of VIADS, without much consideration of artistical aspects or color-blind users. Therefore, this is another limitation of this study. Even though there is no specific feedback on the artistic aspects of VIADS, this can be an area for improvement with appropriate additional expertise in the future.

We aim to increase the impact of VIADS through the (1) promotion of VIADS to increase its visibility among potential users and (2) development of new applications that facilitate the integration of VIADS with electronic health record systems or data repositories. This will enable VIADS to function as an add-on to existing systems that host large amounts of patient data. Through its analytical and visualization capabilities, the integrated version will streamline data sources, thereby promoting the adoption and use of the tool. Increasing the number of terminologies supported by VIADS is another possible area for further investigation. Finally, we could evaluate the tool at various stages and continuously use an iterative design process to improve VIADS.

VIADS, a tool that facilitates the generation of hypotheses in clinical research contexts, is a valuable addition to existing secondary data analysis tools. After intense use sessions, a diverse sample of clinical researchers perceived it to be useful and relatively usable. The new perspectives on hierarchical data sets and an easy-to-use interface provided by VIADS were recognized by users. The availability and use of ICD-9-CM, ICD-10-CM, and MeSH-coded data sets enable practical and convenient comparison of data sets and have many potential health care applications.