For the qualitative analysis, we used framework analysis—a matrix-based method that allows for systematic comparison of themes across cases while retaining case context—to identify both the fundamental skills required of users and the usability barriers that prevented successful use [36]. First, we watched a subsample of video observations to gain a comprehensive understanding of the participants’ experiences and behaviors across the digital tasks. This initial stage helped to identify recurrent ideas, patterns, and areas of interest (typing skills, navigation skills, and human-computer interaction skills). We then developed a thematic framework to guide our analysis by generating a priori coding categories guided by the research questions (“what are the specific and foundational skills a user needs to successfully use digital health tools?” and “what are the usability barriers of these tools?”) and the existing literature about digital literacy and usability [27,32,36-38]. We categorized qualitative codes according to the level of digital literacy they reflected and distinguished them as novice or advanced, where appropriate. This categorization was informed by the digital literacy framework of van Deursen et al [27] and modified to appropriately reflect the skills pertinent to using digital health tools rather than the internet in general. This framework differentiates operational, formal, informational, and strategic digital skills, which capture the range of competencies needed to use digital tools effectively [27]. We primarily focused on operational and formal skills, such as device handling, typing, and navigating digital interfaces, which were most relevant to the tasks in our study. Novice-level skills included novice keyboard skills and novice device skills, which were observed as operational inability or skill gaps when performing a task. Advanced-level skills included advanced typing and advanced device skills—skills that encompassed strategic problem-solving abilities, such as troubleshooting errors and intuitive device understanding. We also mapped the predetermined coding categories of learnability (the capability of digital health tools to enable the user to learn their application) and operability (the capability of digital health tools to enable the user to operate and control them) to challenges associated with completing digital tasks due to usability issues [38]. We also used an inductive coding approach to allow codes to emerge from the data to limit a priori bias. Inductive and deductive coding ensured that both anticipated and novel insights were captured.

Next, we compiled and organized the codes in a structured matrix format, which enabled us to summarize and compare the nature and context of responses, while preserving the richness of the video observations. Each row represented an individual participant, and each column presented the themes associated with a specific task (Figure S1 in Multimedia Appendix 2). As a calibration step, 3 independent coders watched a sample of the videos and created a templated summary for each task in the matrix tool. This was used to assess the template and confirm that the domains were intuitive; identify any missing or incorrectly labeled domains; clarify any issues; and ensure consistency in content, style, and organization across coders. During this process, actions that supported or deviated from these predicted codes led to additional codes, iteratively expanding upon and updating the codebook to include relevant concepts (see Multimedia Appendix 3). After finalizing the codebook and template, to assess coding consistency, 3 members of the research team independently coded 25% of the recordings. After ensuring >85% agreement, the 3 members independently coded the remaining recordings.

For the quantitative component, survey responses were first exported from REDCap for preliminary descriptive summaries, focusing on the variables age, sex, race and ethnicity, preferred language, educational attainment, access to a device with video capability, source of internet access, and enrollment in the SFHN patient portal. These sociodemographic characteristics comprised the covariates for analysis.

Our primary quantitative outcome variable was task completion, which assessed whether a participant independently performed the task only with simple written instructions, performed the task after receiving standard assistance (ie, detailed handouts and video tutorials), performed the task after receiving tailored assistance (ie, verbal instructions and one-on-one help), or was unsuccessful. For analyses, we dichotomized this outcome, categorized as performing a task independently versus requiring any support (standard or tailored) or being unsuccessful (examples of these materials are available online) [39]. This cut point was chosen, as it represents the key distinctions between a user demonstrating fundamental skills and those requiring further assistance or training. Because of technical issues (eg, malfunctioning cameras or recording failures) and participants occasionally declining tasks that they found too difficult, the number of video recordings varied slightly across tasks, as individuals without recordings were treated as missing data and removed from the analysis for that task.

We conducted global tests of association to evaluate relationships between participant sociodemographic characteristics and task completion across all 4 tasks. Kruskal-Wallis tests were used to test the association between age—modeled as a continuous variable—and task completion. χ² tests of independence were used for categorical sociodemographic variables, and Fisher exact tests were used when the expected cell counts were <5. These tests evaluated whether task completion status differed by sociodemographic characteristic when considering all tasks jointly, rather than within a single task. R software (version 4.3.3; R Foundation for Statistical Computing) was used to conduct all analyses.

To assess the relationship between identified themes and task completion, qualitative and quantitative analyses were conducted in parallel using a convergent triangulation design in which the datasets were integrated prior to the combined analysis [40-42]. Key qualitative themes were grouped into higher-order categories to facilitate interpretation. This involved identifying conceptual connections between codes and organizing them into thematic categories. These thematic categories were then converted into categorical variables and incorporated into the quantitative dataset. Triangulating the 2 types of data supported the identification of foundational skills necessary to use digital health tools successfully (and, conversely, the skill gaps that required interventional support). We used χ² tests of independence and Fisher exact tests to determine the significance of associations between each identified skill or skill gap and the task completion outcome. Fisher exact tests were used when the expected cell counts were <5.

The following findings were based on 91% agreement between the 3 coders for the first 15 participants and independent coding of the remaining videos. Thematic findings were grouped into factors, mapped to the overall dimensions of fundamental skills and usability barriers, and reorganized for interpretation. To ease analysis, we decided to separate the codes representing usability barriers from those representing fundamental skills, with several codes representing skills and usability barriers. Figure 1 shows an overview of themes generated during the observational coding process, with subcategories within each key theme, and separated by fundamental skills and usability barriers.

Video observations revealed that participants exhibited a range of digital skills from novice to advanced. The coding matrix categorized participant behaviors into key themes, including typing skills, navigation skills, and human-computer interaction. Typing skills referred to a participant’s ability to navigate the keyboard (particularly capitalizing letters and finding special keys) and demonstration of inefficient typing (“hunting” and searching for letters) or advanced typing skills (proficiency with typing, capitalization, and special keys). Navigation skills comprised inefficient navigation (eg, extraneous scrolling or tapping on the device), novice device skills (eg, unfamiliarity with device layout or apps), advanced navigation and device skills (often indicated by minimal scrolling), and continuous progress of the task. Human-computer interaction (HCI) encompassed codes that demonstrated the way participants engaged or did not engage with their devices to complete the tasks, ranging from how they responded to prompts; whether they were unsure how to proceed, requested assistance, or ineffectively interacted with pop-ups or prompts; to having trouble with specific elements of the tasks (eg, novice URL skills, trouble finding or using a web browser, or novice password skills).

A significant proportion of participants exhibited novice-level digital skills, particularly in typing and navigation. Many participants relied on inefficient typing techniques, such as hunt-and-peck typing and relying on using one finger or a stylus to locate and enter subsequent characters. The inability to appropriately capitalize letters or locate and select special characters created significant barriers to completing the tasks independently. Similarly, many participants demonstrated navigation inefficiencies. Participants exhibited trouble navigating their device with extraneous and repeated tapping or scrolling, confusion locating specific apps or phone settings, or difficulty using a touchscreen or trackpad. In contrast, participants who demonstrated success at navigating the tasks independently almost exclusively displayed advanced navigation skills and the ability to progress through the tasks continuously without breaks in performance flow.

Many individuals also struggled with more complex HCI actions beyond simply navigating their device or using the keyboard. Common observations included participants not understanding how to proceed through the task because they struggled with fundamental concepts such as recognizing common digital icons (eg, settings icon, “close window” button, home icon, or back/forward arrows) or clicking the “next” button, having trouble finding or navigating to a web browser (or conceptually understanding what it is), ineffectively interacting with task pop-ups or prompts (eg, ignoring a pop-up/prompt or choosing the wrong link), and difficulty using the web browser (eg, not knowing how to select the web browser icon to open a new window or how to proceed once the web browser was open). Individuals also demonstrated novice URL and password skills, as they struggled with accurate text entry to successfully type in a URL for a website or enter the provided password. Difficulty performing complex tasks despite receiving instructions was a common reason why participants asked for assistance, often demonstrated even before they attempted the task or after several unsuccessful attempts to complete an action.

Table 3 presents the association between typing, navigation, and HCI codes and task performance. Most notably, across tasks, keyboard skills, navigation skills, and HCI skills showed a significant association between the factor and the ability to complete each task independently. Individuals who demonstrated unfamiliarity with navigating the keyboard, particularly with how to capitalize letters or find special characters, always required assistance or were unsuccessful at completing the digital tasks (P<.001 in all cases). Inefficient navigation with extraneous scrolling or tapping of their device was strongly associated with needing assistance or unsuccessfully completing the task, particularly in task 1 (P=.001), task 2 (P<.001), and task 4 (P=.006), which required more complex device navigation to complete the task. Novice device skills, demonstrated by lack of familiarity with one’s device, whether it be the layout, finding and using apps, or navigating the trackpad, were significantly associated with requiring assistance or unsuccessfully completing the task across tasks (task 1, P<.001; task 2, P<.001; task 3, P=.004; task 4, P=.002). Typing skills (both inefficient typing and advanced typing skills) were not significantly associated with digital task completion (exact P values given in Table 3). However, we did not assess speed to completion.

The complexity of the digital tasks required several fundamental human-computer interaction skills, beyond navigating one’s device, to complete each task. Displaying confusion or hesitation with how to proceed during a task was a significant barrier for individuals during each task, ranging from not being sure how to proceed to confusion about how to navigate barriers that arise or where to put information (tasks 1, 2, and 4, P<.001; task 3, P=.02). Individuals also often asked for assistance when stuck on a task, demonstrating the complexity of progressing through a task as a barrier to independently completing a task (P<.001 in all cases). In task 2, the ability to find and use a web browser appropriately and the ability to type in a URL correctly were also significantly associated with being unsuccessful or requiring assistance to complete the task (P<.001 for both). In tasks 3 and 4, having trouble with creating a username or password (eg, not sure how to create one) was associated with needing assistance or unsuccessfully completing the task (task 3, P=.004; task 4, P=.006).

Video observations also revealed that task completion was hindered by several usability barriers. The key usability barriers across all the tasks were related to learnability and operability. Tables 4 and 5 summarize the learnability and operability issues for each task. In task 1, joining a video visit from a text message and unfamiliarity with system-generated notifications led to hesitation or errors in navigating the log-in prompt (characterized by the codes “confusion/hesitation regarding text messages” and “unsure how to proceed”), illustrating learnability barriers to joining video visits, despite the instructions in the text. In addition, individuals often did not select the video link, instead opting to choose an alternative method, such as replying to the text (despite it being a no-reply message) or attempting to join the video visit unsuccessfully. Additionally, once the participant successfully selected the link in the text message, additional pop-ups and links necessary to join the video call confused the participant, resulting in those pop-ups and links being ignored or ineffectively addressed. The operability limitation of this task was that the system did not provide clear enough cues for individuals to correctly select the link to join the video visit; therefore, individuals often did not know which link to select and instead selected the wrong link, either delaying their ability to join a video visit or preventing it altogether.

Task 2 was the most difficult for participants, with learnability and operability barriers affecting the ability to complete the task of signing in to the patient portal on a web interface. Many participants displayed misinterpretation of browser suggestions and confusion regarding where to type in the URL (coded as “mistake search bar for URL bar”) and the extent to which the URL needs to be complete (categorized as “how to do manual credential entry;” many stopped before typing in the complete URL or mistakenly used autofill to create a false URL), illustrating the learnability barriers associated with this task. In addition, individuals were often unsure how to proceed, sometimes selecting irrelevant links or failing to recognize the correct link to continue to the patient portal website. Typing in a URL with multiple characters and capitalizations also proved to be a user burden, as many individuals made small mistakes that led to repeated failures, without indications of where the error was, illustrating the operability barrier in URL design. These mistakes included typos in the URL or using autofill to automatically complete the URL, despite the autofill not matching the provided URL.

Participants struggled with the password aspects of tasks 3 and 4, further exposing learnability and operability barriers. They often misunderstood the login prompts, illustrating conceptual clarity issues in password duplication or creation for digital tools (categorized as “how to do manual credential entry” and “how to create a password”). The knowledge required to understand and replicate the complexity required for password creation is too steep, resulting in learnability barriers for many users who do not possess the basic level of knowledge necessary to navigate username and password duplication or creation. Additionally, in task 4, once individuals created a password, they often did not know what to do next to proceed, illustrating the learnability issues of progressing successfully through the steps required to sign in or sign up for a patient portal account. The operability limitation of these tasks was notifications that did not adequately notify the participant of their error, leading to repeated errors (categorized as “generic error notification leads to repeated errors” and “‘login failed’ notification leads to repeated errors”). For example, the activation codes were confusing and led to errors among participants without proper indication. When prompted that an activation code was needed, many did not know what it was, where to find it, or what to do with it. If they mistyped the activation code but not the username or password, they would receive only a “login failed” notification that did not properly indicate where the error occurred.

Nearly half of the participants were able to complete each task independently, with 56.3% (36/64) able to complete tasks 1, 3, and 4 independently. The task of typing in a URL in task 2 posed unique barriers and resulted in more participants requiring assistance or being unsuccessful. This is potentially explained by the fact that typing in a complete URL is increasingly uncommon in digital health contexts, where users typically rely on search engines (eg, searching “SFHN MyChart patient portal” in the search bar), use autocomplete based on browsing history, or access the portal through an app. This is further supported by the fact that many participants struggled to recognize the URL bar or conceptually understand what was being asked of them during task 2. Despite this, many informational handouts about the patient portal continue to direct users to type in a specific URL to access the patient portal. These materials should consider providing alternative instructions that align with common user behavior, such as using search terms or app-based access.

We anticipated that there would be disparities in task completion based on age, sex, race and ethnicity, preferred language, education, access to a device with video capability, source of internet access, and patient portal enrollment. Older age, minoritized races and ethnicities, preferred languages other than English, low educational attainment, possession of a device without video capability, data as the source of internet access, and not being enrolled in the patient portal contributed significantly to gaps in digital skills. These sociodemographic characteristics were significantly associated with requiring assistance or being unsuccessful, particularly for tasks that involved multiple steps or complex engagement with digital interfaces. This suggests that individuals from these groups may be disproportionately less likely to have the skills required to feel confident and able to complete each task independently. Moreover, tasks 3 and 4 were more complex, involving reading comprehension and following multiple prompts to enter pertinent text or create a unique password. These elements (such as log-in prompts and password creation details) were in English only, which posed obvious barriers to individuals who preferred non-English languages.

Although this study is not sufficiently powered to examine the intersectional vulnerabilities of participants who experience multiple, overlapping socioeconomic disadvantages—factors that may confound their ability to complete digital tasks independently—there remains a critical need for future research to adopt an inclusive approach to participant recruitment and analysis in usability studies. Specifically, examining how compounded forms of marginalization (eg, low income, limited education, language barriers, and age-related factors) shape digital engagement is essential for uncovering the full extent of challenges faced by individuals most at risk of digital exclusion. Such efforts are key to informing the design of equitable digital health tools and implementation strategies that promote broader and more effective adoption across diverse populations [17,21,22,33,44].

These results illustrate that the basic foundational skills necessary for successful navigation and use of digital health tools include competency with one’s device, such as knowing basic keyboard navigation and how to find appropriate apps or change settings, being able to navigate complex processes such as interacting with pop-ups and successfully navigating web browsers, and being able to proceed after each step. Without these basic foundational skills, an individual’s ability to successfully use and navigate digital health tools decreases significantly. Strategies for improving digital skills to ensure that individuals have the necessary foundational skills include implementing digital competency screening tools and providing digital skills workshops and trainings that emphasize hands-on practice with common tasks such as manual text entry, device and keyboard navigation, managing settings, and interacting with complex pop-ups or prompts. Prior work from our team has demonstrated that the brief, 9-item Skills Measurement and Readiness Training for Digital Health (SMART Digital Health) Scale is reliable for measuring skills for digital health-related tasks and activities and predicting the level of support an individual may need [23].

In addition, prior studies have found that providing support such as technical support or digital education alleviates some of the digital proficiency challenges faced by many and promotes the use of digital health tools, particularly among those who face greater barriers to digital use [45-47]. In addition to providing detailed assistance and training to address the complex barriers faced by those with limited digital literacy, ongoing or repeated support may be necessary, as one-time training or technical help may not be sufficient to overcome persistent technical challenges [47].

The findings highlight the critical role of learnability and operability challenges across a range of digital health tasks commonly required for digital patient engagement. Participants encountered distinct barriers during each task, verbalizing frustration and making repeated assistance requests. Many of the difficulties were often tied to misunderstandings about how to initiate or complete a task, suggesting that digital inclusion efforts should simplify interfaces, provide clear prompts, and reduce extraneous steps to mitigate the usability issues faced by many participants.

Learnability issues can lead to a range of negative user experiences, including hesitation and confusion, a greater need for assistance and additional support to complete each task, and low user confidence. Each task required a level of understanding that users might not have to complete the task, resulting in a system that does not equally support users of all backgrounds to adapt and use the system. In addition, there were operability barriers across each task. Particularly, difficulties such as clicking the wrong link, making errors without descriptive or specific notifications, or being confused about how to proceed during a task result in a system that does not provide users with the necessary functionalities to allow users of all backgrounds to adapt and use the system. Efforts to improve the clarity of digital task prompts and processes, ranging from simplifying the URL to log in to the patient portal to clarifying how to successfully join a video visit from a text (or link) or how to engage with certain prompts, will enhance system learnability and operability and allow users to better engage with the system and remember how to use the system going forward.

These findings are consistent with prior research identifying digital navigation and account access as common friction points for users with limited digital literacy [9,11]. However, our human-centered design (HCD) approach to this study adds nuance by documenting how these challenges manifest in real time and vary by task complexity. Previous literature often examines digital competence in aggregate, but our task-specific analysis reveals that even users with some experience may struggle with unfamiliar or multistep processes, particularly when there are long task completion times or when confusing interfaces or prompts lead to repeated errors [48,49]. As adoption of digital health tools continues to increase, there are opportunities to use HCD approaches to identify usability barriers and align digital health tool design with user needs to ensure a more equitable distribution of digital health tool benefits [32]. This is particularly important, as there are numerous examples in which digital tools exacerbate inequities. HCD approaches can help uncover previously unknown health equity issues and drive efforts to address those inequities more acutely experienced by marginalized populations that are often not considered priorities in research [28-30,32,33].

The findings from this study underscore the pressing need for digital health vendors to take a more active role in addressing usability barriers and skill gaps through cocreation. While health systems increasingly rely on digital platforms to facilitate patient engagement, results demonstrate that usability challenges and complex design, often resulting from a failure to cocreate with users, are common friction points that inhibit successful digital health tool use. These barriers are not merely technical inconveniences; they are structural barriers that prevent users from accessing care and support. Prior studies have similarly identified usability challenges—such as manually entering data, following procedures to retrieve data (such as appointment details), and understanding medical terminology—as key barriers that prevent equitable use of patient portals among diverse populations [49,50].

To address usability challenges, vendors must adopt HCD principles and co-design strategies that engage diverse users throughout the design process to ensure that digital interfaces are intuitive, accessible, and responsive to users’ varying levels of digital proficiency. This includes incorporating plain language, reducing unnecessary steps, improving language functionalities for non-English speakers, and reducing privacy and security complexities that act as a barrier to users with low digital literacy [50]. Vendors must not assume a baseline level of digital literacy and should instead actively test for, design against, and consult patients about learnability, operability, and task complexity barriers in these digital health tools. By treating usability and accessibility as core priorities rather than downstream implementation concerns, vendors can play a pivotal role in promoting digital equity and closing gaps in health care access. For example, strict security needs for usernames and passwords (including activation codes and other 2-factor authentication) are not “user-friendly,” and while they make sense from a security perspective, they are advanced skills that many users do not understand or have experience with. Issues with complex password creation, entering an activation code, and other 2-factor authentication methods are demonstrated tension points for users from marginalized backgrounds who may have little experience in engaging with strict and complex privacy protocols. Vendors must include patient voices when weighing privacy concerns against usability to ensure that access and usability decisions do not further inequities [28,50].

In the meantime, without more usable products, health care systems need to be prepared to support patients in accessing and using existing digital health tools. As prior studies have shown, explicit identification of a “digital navigator” to support patients in using digital health tools facilitates equitable access and use of digital health tools [51,52]. Unfortunately, this technical assistance often falls to clinical care team members who are not trained to provide this type of support; as a result, patients do not receive adequate support, particularly in safety net systems [52,53]. Increasing support for digital navigator training and incorporating them as members of the clinical care team will be crucial to help overcome gaps in foundational skills and usability barriers.