This systematic review was reported in accordance with the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) statement [21]. The synthesis methods were guided by the Cochrane Handbook guidance for synthesis without meta-analysis [22]. A review protocol was not registered, as this review began as an exploratory assessment of how literacy and portal outcomes were measured.

We included peer-reviewed articles and gray literature, including conference abstracts and papers, as well as theses, to minimize publication bias. Eligible studies (1) involved adults aged 18 years or older from clinical, community, or online samples; (2) evaluated a real-world patient portal, defined as a portal implemented in routine health care and used by patients rather than a hypothetical or simulated portal; (3) measured health literacy or eHealth literacy using a validated instrument; (4) reported a quantitative association between health literacy or eHealth literacy and patient portal access or use; and (5) were published in English. We excluded studies that evaluated nonportal technologies, such as electronic health record access without a patient portal, or focused only on user perceptions or qualitative outcomes, such as satisfaction, usability, or ease of use.

We conducted a systematic search in PubMed, Scopus, CINAHL, PsycINFO, Web of Science, and Embase using database-specific free-text search strings related to health literacy, eHealth literacy, patient portals, and portal access or use, combined with Boolean operators. The search strategy was developed by the research team without support from a research librarian. Full search strings for each database are provided in Multimedia Appendix 1. No restrictions were applied to the publication date range. All searches were conducted on March 20, 2026.

All search results were imported into Covidence, where duplicates were identified and removed automatically or manually before title and abstract screening. Five reviewers (PH, MA, AP, AR, and SK) screened titles and abstracts in Covidence using predefined eligibility criteria, with each record assessed independently by 2 of them. Studies that passed the initial title and abstract screening advanced to full-text review, where the same 5 reviewers (PH, MA, AP, AR, and SK) applied the eligibility criteria, again with each article assessed independently by 2 of them. Any disagreements were resolved through discussion and consensus.

Interrater agreement for title and abstract screening, measured with Cohen's kappa (κ), ranged from κ=0.50 to κ=0.57, indicating moderate agreement [23]. The moderate kappa values likely reflect κ’s prevalence sensitivity, since most title and abstract screening decisions were exclusions, which can lower kappa despite high observed agreement [24,25]. In contrast, proportion agreement for title and abstract screening ranged from 85% to 97%. Discrepancies arose from unclear abstracts, especially when studies did not clearly distinguish real-world patient portals from hypothetical portals or other digital technologies or did not clearly report quantitative outcomes of portal access or use. For full-text screening, agreement ranged from κ=0.79 to κ=1.00, indicating substantial or almost perfect agreement.

For each study, we extracted data on key study characteristics, literacy measurement, and patient portal outcomes. Two of the 5 reviewers (PH, MA, AP, AR, and SK) independently extracted and compared data for each study, and any discrepancies were resolved by consensus. Interrater agreement for data extraction was substantial, with κ ranging from 0.76 to 0.80.

To support consistent synthesis across heterogeneous studies, we developed an outcome classification scheme during data extraction and applied it across all included studies. The framework and its definitions were derived inductively from the portal outcomes reported in the included studies, refined through reviewer discussion, and then applied consistently across studies. Specifically, we classified outcomes at 2 levels: outcome domain and portal outcome. The outcome domain referred to the broad stage of portal engagement, either portal access or portal use. We defined portal access as the early steps required to gain entry to the portal, and portal use as behaviors occurring after access had been established. The portal outcome referred to the specific action within each domain. This process yielded the following hierarchy:

1. Domain 1: portal access

2. Domain 2: portal use

Following Beal et al [18] suggestions for quantifying portal use, we also classified portal use outcomes by outcome measure. Within the portal use domain, outcome measures were classified as (1) any use, defined as a binary indicator of whether a patient used the portal overall or a specific feature at least once, or (2) frequency of use, reflecting how often the patient used the portal or feature.

We first identified whether each study measured health literacy or eHealth literacy. Most health literacy studies used dichotomous categorical measures, though the labels varied across studies, for example, limited, low, problematic, inadequate, or no limitation. To improve comparability across studies, we harmonized all categorical classifications to a common lower versus higher health literacy framing. A few studies used 3 ordered categories, such as limited, marginal, and adequate health literacy. In those cases, we used the most extreme contrast, for example, limited versus adequate health literacy, to align results with the lower versus higher framing. The original comparison framing and specific thresholds used across studies are shown in Multimedia Appendix 2 for reference. Studies that treated health literacy as a continuous variable were also interpreted directionally, from lower to higher health literacy. We applied the same approach to eHealth literacy. Most eHealth literacy studies used continuous measures, which were interpreted directionally from lower to higher eHealth literacy, whereas categorical classifications were harmonized to the same lower versus higher framing.

To assess risk of bias, we used the Joanna Briggs Institute (JBI) critical appraisal tools matched to the study designs, including the analytical cross-sectional [26], the cohort [27], and the quasi-experimental checklists [28]. We restricted the appraisal to full-text, peer-reviewed journal articles. Each criterion was rated yes, no, or unclear. Two of the 5 reviewers (PH, MA, AP, AR, and SK) assessed each study independently and resolved disagreements through consensus discussion. Interrater agreement for risk-of-bias items yielded κ values ranging from 0.63 to 0.75, indicating substantial agreement.

The synthesis first summarized study characteristics descriptively and quantified the number of studies that examined each portal outcome, outcome measure, and literacy type.

Because studies differed substantially in the literacy instruments, portal outcomes, and outcome measures, we judged a pooled quantitative synthesis inappropriate. Instead, we used direction-based vote counting to summarize the direction of associations between health literacy or eHealth literacy and patient portal outcomes, regardless of effect size magnitude and statistical significance [22].

To account for potential confounding, we prioritized fully adjusted estimates when multiple estimates were reported. Effects showing no difference were coded as “no association” and included as a separate category. Some studies reported only statistical significance without effect estimates, whereas others reported descriptive findings only. In these cases, effects were considered noncodable because no clear directional comparison could be derived from the reported results. Studies reporting partially tested findings without a clear direction, or descriptive findings only, were therefore excluded from direction-based coding and summarized using narrative synthesis [29]. Multimedia Appendix 3 presents the study-level effect estimates, directional classifications, and reasons for exclusion from directional coding.

For studies included in direction-based vote counting, we followed the recommended practice of displaying effect directions visually in an effect direction plot [30]. For visual display in the effect direction plot, we grouped studies into smaller (<500 participants) and larger (≥500 participants) samples to provide a simple visual indication of study size across a heterogeneous evidence base.

Data synthesis was conducted at the level of one effect per study for each predefined portal outcome. A single study could contribute multiple effects if it reported different portal outcomes. To maintain a consistent orientation across studies, all effect estimates were standardized to the lower literacy perspective. For example, health literacy estimates originally reported for higher versus lower health literacy were inverted so that all coded effects reflected lower versus higher health literacy. The same approach was applied to eHealth literacy.

We decided not to assess the certainty of the evidence because the synthesis was based on direction-based vote counting rather than pooled effect estimates, which limits the assessment of certainty domains that rely on comparable effect estimates and CIs across studies.

A total of 2347 records were identified through database searches. After removal of 1240 duplicates, 1107 unique records remained for title and abstract screening. Of these, 105 records advanced to full-text review. After full-text assessment, 75 records were excluded, leaving 30 studies [31-60] for data extraction and synthesis. The full study selection process is shown in Figure 1.

Most included studies were journal articles (26/30, 87%) [33-39,41-45,47-60], while 2 were conference abstracts (2/30, 7%) [31,32], one was a conference paper (1/30, 3%) [40], and one was a thesis (1/30, 3%) [46]. Most studies were conducted in the United States (24/30, 80%) [32,34-43,46-50,52-57,59,60] and used a cross-sectional design (23/30, 77%) [31,34-37,39,41,43-51,53-59]. Sample sizes ranged from 18 to 60159, with most samples drawn from adult clinical populations (24/30, 80%) [31-33,35-47,50,52-57,60]. Survey self-report was the most common portal data source (21/30, 70%) [31,33-37,39,41,43-46,48-51,53-55,58,59], whereas portal log data were used in 8 studies (27%) [32,38,40,42,47,52,57,60] and administrative data in one study (3%) [56]. Health literacy alone was examined in 17 studies (57%) [32-34,36-40,42,47,49,52,55-58,60], eHealth literacy alone in 10 studies (33%) [41,44-46,48,50,51,53,54,59], and both literacy constructs in 3 studies (10%) [31,35,43]. The most common health literacy instruments were the Brief Health Literacy Screener (BHLS) [34,36-40,42,43,52,56] and the Newest Vital Sign (NVS) [35,47,49,55,57,60], whereas eHealth literacy was measured most often with the eHealth Literacy Scale (eHEALS) [35,41,43,45,46,48,50,51,53,54]. Health literacy was usually treated as a categorical variable, most often through dichotomous or ordinal groupings, although thresholds varied across studies and were not always reported. By contrast, most studies using eHEALS treated it as a continuous measure, while 2 studies applied dichotomous eHEALS cutoffs (Multimedia Appendix 2). Portal types included mainly institution-specific portals (25/30, 83%) [32-42,45-54,56,57,59,60], with the remainder using national patient portals (5/30, 17%) [31,43,44,55,58]. Table 1 summarizes the characteristics of all included studies.

Risk of bias was appraised for the 26 journal papers [33-39,41-45,47-60]. Across cross-sectional studies, most JBI criteria were met. The most common risk-of-bias concern was the measurement of patient portal use through self-report rather than objective system data, which affected the reliability of outcome measurement in the majority of studies.

Across cohort studies, most criteria were also met. The most common risk-of-bias concern was inadequate handling of follow-up.

The single quasi-experimental study had risk-of-bias concerns related to the absence of a control group, baseline differences between user groups, a single pre-post time point, incomplete follow-up, and ceiling effects on half of the outcome measures [33]. Risk-of-bias assessments for all criteria and each study are presented in Multimedia Appendix 4.

The aim of this review was to synthesize associations of health literacy and eHealth literacy with patient portal access, general use, feature-specific use, and frequency of use. Several main findings were identified. In our review, the evidence base was uneven: health literacy was examined more often than eHealth literacy, included studies focused primarily on general portal use rather than portal access and specific features, and studies relied mainly on any-use measures rather than frequency of use. Despite these imbalances, health literacy and eHealth literacy may not function interchangeably as predictors of portal engagement. Nearly every directionally codable eHealth literacy study reported lower general portal use among patients with lower eHealth literacy. Health literacy showed a more mixed pattern for general use. The 3 studies that assessed both constructs within the same sample mostly supported this contrast, although they remain too few for firm comparative inference. In contrast, lower health literacy was more often associated with lower portal access and feature-specific use, but comparable eHealth literacy evidence for those outcomes remained sparse.

In our review, studies focused predominantly on general portal use, while early engagement stages, such as registration and first login, as well as specific portal features, received less attention. One possible explanation for this pattern is that many included studies recruited participants who were already registered portal users and therefore excluded the access stage from analysis. Another plausible explanation is that overall portal use is easier to measure than feature-specific behaviors, especially in cross-sectional surveys. Yet, treating portal use as a single outcome oversimplifies the staged nature of real-world portal engagement [61]. Access and subsequent portal use represent distinct stages at which disparities may emerge, and access alone does not necessarily translate into continued use [56,57,62]. Studies that measure only overall use may therefore miss whether literacy-related barriers arise during initial access, continued use, or both.

Across the included studies in our review, health literacy was examined more often than eHealth literacy. This imbalance likely reflects the longer history of health literacy as a field. Health literacy became central to health communication research [63] and was originally conceptualized in the context of paper-based health resources [12], well before digital health information became a major focus. Its recognition by the US Department of Health and Human Services further reflects its prominence at the policy level [64]. eHealth literacy, in contrast, was formally introduced only in 2006, as health information seeking increasingly moved into digital environments [12]. Prior digital health research appears to show a similar pattern, with greater emphasis on health literacy than on eHealth literacy [65]. This imbalance matters. Using health literacy as a proxy for eHealth literacy risks misclassifying digital navigation problems as comprehension problems, which may lead health systems to offer the wrong type of support.

A further pattern in our findings was the stronger emphasis on any use than frequency-based outcomes. This pattern is consistent with broader patient portal research, which often reports portal adoption or use without examining how frequently portals are used over time [18]. In our review, most studies relied on cross-sectional surveys to measure portal engagement. Although surveys are appropriate for capturing behavior at a single time point, they are less suited to assessing temporal patterns. Because these designs rely primarily on self-reported rather than objective data, they introduce recall bias [66,67] and provide limited information about when portal use occurred or how often it was repeated. Longitudinal designs using portal audit logs and defined follow-up periods are better suited to capturing repeated use over time [42]. As a result, the literacy evidence base reflects initial uptake more than sustained engagement and may underestimate barriers that emerge only with continued use.

Beyond outcome measurement, the choice of literacy constructs, instruments, and modeling approaches may have contributed to the observed patterns. One possible explanation is conceptual. Health literacy and eHealth literacy instruments differ substantially in what demands within the portal experience they measure. Across included studies in this review, eHealth literacy was most often assessed with eHEALS, a self-report measure of perceived ability to find, appraise, and use online health information [68]. Health literacy was more often assessed with BHLS, which focuses on perceived difficulty with written clinical materials [69], or NVS, a performance-based test of functional comprehension and numeracy [70]. These differences suggest, but do not establish, that the 2 constructs may align with different aspects of portal engagement. For example, eHealth literacy may be more relevant during digital uptake and navigation, before patients reach portal content, whereas health literacy may become more relevant once patients must interpret and act on test results, clinical notes, or medication information. Our findings offer partial support for this interpretation, as lower health literacy showed more consistent associations with feature-specific portal outcomes than with general portal use. Comparable evidence for eHealth literacy at the feature level, however, remains too limited to allow a firm conclusion. At the same time, the observed difference may not be entirely conceptual and may instead reflect measurement choices. In our review, most studies modeled eHealth literacy as a continuous variable, while health literacy measures were often categorized. Because categorizing a continuous measure reduces information and statistical power [71,72], health literacy associations may have appeared weaker than the underlying relationship. This problem is further compounded by variations in the health literacy thresholds used across studies, with some studies not reporting thresholds at all. The 3 studies that assessed both literacy constructs within the same sample were broadly consistent with the overall pattern; however, they did not simultaneously model both in adjusted analyses. Given this limited within-sample evidence, it is unclear whether health literacy’s weaker associations reflect true differences, confounding, measurement choices, or both.

Given the uncertainty about whether the observed differences between health literacy and eHealth literacy are due to true construct differences, confounding, or both, future portal studies should measure both constructs simultaneously within the same sample using comparable approaches. Modeling both constructs continuously, where appropriate, would preserve statistical power and allow more direct comparison across portal outcomes. Studies should also distinguish portal access from portal use and separate any use from frequency-based outcomes to improve comparability across initial uptake and sustained engagement.

For clinicians and health systems, the choice of literacy measure can help identify the type of portal barrier a patient faces and guide more targeted intervention design. To improve registration, first login, or basic navigation, digital navigation barriers may warrant greater attention. If the goal is to improve understanding and use of portal content after login, health literacy-sensitive communication strategies may be more relevant. For health systems, portal equity assessment should extend beyond account creation or activation. Our findings suggest that literacy-related disparities may persist after access is established, especially for health literacy and feature-specific use, which could go undetected by systems that track only registration or activation. A patient may appear engaged in activation metrics yet never progress to viewing test results, managing medications, or messaging clinicians, the features through which portals affect care. These channels carry abnormal results, medication changes, and clinician messages about findings that change care. The 2 oncology studies in our sample found lower engagement in exactly these channels among patients with lower health literacy [32,52]. As measurement consistency improves, future studies should be better positioned to identify which literacy construct is most informative for specific portal barriers.

This review has methodological limitations. First, we used direction-based vote counting. This approach ignores the magnitude of the effect size and gives equal weight to all studies regardless of study quality or risk of bias. Second, most comparisons between health literacy and eHealth literacy were indirect, because only 3 studies assessed both constructs in the same sample. Third, we harmonized heterogeneous literacy measures into a lower-versus-higher framing. This simplification may have obscured differences captured by 3-level categorical distinctions, such as limited, marginal, and adequate health literacy. Fourth, studies often modeled health literacy and eHealth literacy differently, with eHealth literacy typically treated as a continuous variable and health literacy often categorized, further limiting direct comparisons between the 2 constructs. Fifth, the outcome classification framework was developed inductively and was not prespecified, which may have introduced some subjectivity in how outcomes were grouped. Sixth, the review was limited to English-language studies, which may have introduced language bias. Seventh, a review protocol was not registered beforehand, which limits the transparency of the prespecified review methods. Eighth, searches relied on free-text terms and did not include database-specific vocabulary, such as MeSH terms and CINAHL subject headings, which could have decreased search sensitivity. Ninth, manual reference-list or citation searching was not conducted, which may have omitted relevant studies that were not identified through database searches. Finally, because no publication date restriction was applied, some older studies may have evaluated patient portals with different user interfaces and functionalities, which could have affected how directly their findings apply to contemporary portal use.

Limitations identified in the included literature should also be noted. Most studies were conducted in the United States, relied on clinical populations, and evaluated institution-specific portals, which limits generalizability to national portals and community populations. Differences in literacy measures, thresholds, and outcome measures (eg, self-reported use vs portal logs), as discussed above, further limit direct cross-study comparisons.

Current evidence points to a clear equity problem in patient portal engagement. Patient portals are now central to care, but research still relies on broad measures, most often general portal use or binary any-use outcome measures. This approach risks missing where literacy-related disparities emerge. Across the available evidence, lower eHealth literacy showed a more consistently negative cross-study pattern for general portal use, whereas lower health literacy more often showed negative associations with portal access and several feature-specific outcomes. Because only 3 studies assessed both constructs within the same sample, this contrast should be interpreted as a cross-study pattern rather than a direct comparative finding.

Future research should move beyond broad portal-use measures and examine where disparities occur across the full continuum of portal engagement. Studies should measure both health literacy and eHealth literacy in the same sample, use objective portal log data when possible, and distinguish access, any use, frequency, and feature-specific behaviors. This shift would help health systems identify which patients are left behind after portals become available and design support matched to the barrier: digital navigation, clinical comprehension, or both.